2024-03-29T11:08:01.734 INFO:root:teuthology version: 0.0.1.dev262+ge691533 2024-03-29T11:08:01.734 DEBUG:teuthology.run:Teuthology command: teuthology --archive /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622699 --description rados:verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-snappy rados tasks/rados_api_tests validater/lockdep} --owner scheduled_amathuri@teuthology --verbose --name amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi -- /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622699/orig.config.yaml 2024-03-29T11:08:01.769 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-29T11:08:01.854 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622699 branch: main description: rados:verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-snappy rados tasks/rados_api_tests validater/lockdep} email: null first_in_suite: false job_id: '7622699' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 10 - volumes: count: 4 size: 10 os_type: centos os_version: 9.stream overrides: admin_socket: branch: main ceph: conf: client: debug monc: 20 debug ms: 1 debug objecter: 20 debug rados: 20 global: lockdep: true mon client directed command retry: 5 mon election default strategy: 1 ms bind msgr2: false ms inject socket failures: 5000 ms type: async mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon min osdmap epochs: 50 mon osdmap full prune interval: 2 mon osdmap full prune min: 15 mon osdmap full prune txsize: 2 mon scrub interval: 300 mon warn on pool no app: false paxos service trim min: 10 osd: bluestore block size: 96636764160 bluestore compression algorithm: snappy bluestore compression mode: aggressive bluestore fsck on mount: true bluestore zero block detection: true debug bluefs: 20 debug bluestore: 20 debug monc: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd class default list: '*' osd class load list: '*' osd client watch timeout: 120 osd debug verify cached snaps: true osd debug verify missing on start: true osd failsafe full ratio: 0.95 osd mclock override recovery settings: true osd mclock profile: high_recovery_ops osd objectstore: bluestore osd op queue: debug_random osd op queue cut off: debug_random osd shutdown pgref assert: true flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(OSD_SLOW_PING_TIME - reached quota - overall HEALTH_ - \(CACHE_POOL_NO_HIT_SET\) - \(POOL_FULL\) - \(SMALLER_PGP_NUM\) - \(SLOW_OPS\) - \(CACHE_POOL_NEAR_FULL\) - \(POOL_APP_NOT_ENABLED\) - \(PG_AVAILABILITY\) - \(OBJECT_MISPLACED\) - slow request mon_bind_msgr2: false sha1: 5dcb190da270440ef141d088eb27a9fdb8afc5f2 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} install: ceph: flavor: default sha1: 5dcb190da270440ef141d088eb27a9fdb8afc5f2 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: main sha1: 1fb5d10608dccee73edd6ef8a2c7b804a097c324 owner: scheduled_amathuri@teuthology priority: 150 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mon.c - mgr.y - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - node-exporter.a - - mon.b - mgr.x - osd.4 - osd.5 - osd.6 - osd.7 - client.1 - prometheus.a - node-exporter.b seed: 8152 sha1: 5dcb190da270440ef141d088eb27a9fdb8afc5f2 sleep_before_teardown: 0 subset: 99/1000 suite: rados:verify suite_branch: main suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph-ci.git suite_sha1: 1fb5d10608dccee73edd6ef8a2c7b804a097c324 targets: smithi084.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBEHXNFai/jEmct9ZIBZFZkJRhsklU2T6DhlQYD7OOFnU/7MBtkG1eum5vdqLyZuTMzqVORCX+BgUC7grhC+ksv4= smithi179.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPL2qn7wt1ESWLkdih7XgUUtWDFzamLA5GFZpkPo13TiiE7jP24czk2rm6vPwLHGZeXugXdNWG/bPvTWDQXA12I= tasks: - install: null - ceph: null - workunit: clients: client.0: - rados/test.sh env: ALLOW_TIMEOUTS: '1' timeout: 6h teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: e691533f9cbb33d85b2187bba20d7102f098636d timestamp: 2024-03-26_05:45:44 tube: smithi user: amathuri verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-03-29T11:08:01.854 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa; will attempt to use it 2024-03-29T11:08:01.855 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks 2024-03-29T11:08:01.855 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-03-29T11:08:01.856 INFO:teuthology.task.internal:Checking packages... 2024-03-29T11:08:01.879 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '5dcb190da270440ef141d088eb27a9fdb8afc5f2' 2024-03-29T11:08:01.879 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-03-29T11:08:01.879 INFO:teuthology.packaging:ref: None 2024-03-29T11:08:01.879 INFO:teuthology.packaging:tag: None 2024-03-29T11:08:01.880 INFO:teuthology.packaging:branch: main 2024-03-29T11:08:01.880 INFO:teuthology.packaging:sha1: 5dcb190da270440ef141d088eb27a9fdb8afc5f2 2024-03-29T11:08:01.880 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=main 2024-03-29T11:08:02.154 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-2530.gb74786fc 2024-03-29T11:08:02.155 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-03-29T11:08:02.162 INFO:teuthology.task.internal:no buildpackages task found 2024-03-29T11:08:02.162 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-03-29T11:08:02.216 INFO:teuthology.task.internal:Saving configuration 2024-03-29T11:08:02.229 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-03-29T11:08:02.254 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-29T11:08:02.279 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi084.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622699', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-29 11:02:10.786447', 'locked_by': 'scheduled_amathuri@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBEHXNFai/jEmct9ZIBZFZkJRhsklU2T6DhlQYD7OOFnU/7MBtkG1eum5vdqLyZuTMzqVORCX+BgUC7grhC+ksv4='} 2024-03-29T11:08:02.303 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi179.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622699', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-29 11:02:10.788354', 'locked_by': 'scheduled_amathuri@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPL2qn7wt1ESWLkdih7XgUUtWDFzamLA5GFZpkPo13TiiE7jP24czk2rm6vPwLHGZeXugXdNWG/bPvTWDQXA12I='} 2024-03-29T11:08:02.303 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-03-29T11:08:02.310 INFO:teuthology.task.internal:roles: ubuntu@smithi084.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2024-03-29T11:08:02.310 INFO:teuthology.task.internal:roles: ubuntu@smithi179.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2024-03-29T11:08:02.310 INFO:teuthology.run_tasks:Running task console_log... 2024-03-29T11:08:02.424 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f75a17190d0>, signals=[15]) 2024-03-29T11:08:02.424 INFO:teuthology.run_tasks:Running task internal.connect... 2024-03-29T11:08:02.432 INFO:teuthology.task.internal:Opening connections... 2024-03-29T11:08:02.432 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi084.front.sepia.ceph.com 2024-03-29T11:08:02.434 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi084.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T11:08:02.526 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi179.front.sepia.ceph.com 2024-03-29T11:08:02.527 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi179.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T11:08:02.616 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-03-29T11:08:02.624 DEBUG:teuthology.orchestra.run.smithi084:> uname -m 2024-03-29T11:08:02.642 INFO:teuthology.orchestra.run.smithi084.stdout:x86_64 2024-03-29T11:08:02.642 DEBUG:teuthology.orchestra.run.smithi084:> cat /etc/os-release 2024-03-29T11:08:02.699 INFO:teuthology.orchestra.run.smithi084.stdout:NAME="CentOS Stream" 2024-03-29T11:08:02.699 INFO:teuthology.orchestra.run.smithi084.stdout:VERSION="9" 2024-03-29T11:08:02.699 INFO:teuthology.orchestra.run.smithi084.stdout:ID="centos" 2024-03-29T11:08:02.699 INFO:teuthology.orchestra.run.smithi084.stdout:ID_LIKE="rhel fedora" 2024-03-29T11:08:02.700 INFO:teuthology.orchestra.run.smithi084.stdout:VERSION_ID="9" 2024-03-29T11:08:02.700 INFO:teuthology.orchestra.run.smithi084.stdout:PLATFORM_ID="platform:el9" 2024-03-29T11:08:02.700 INFO:teuthology.orchestra.run.smithi084.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-29T11:08:02.700 INFO:teuthology.orchestra.run.smithi084.stdout:ANSI_COLOR="0;31" 2024-03-29T11:08:02.700 INFO:teuthology.orchestra.run.smithi084.stdout:LOGO="fedora-logo-icon" 2024-03-29T11:08:02.700 INFO:teuthology.orchestra.run.smithi084.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-29T11:08:02.700 INFO:teuthology.orchestra.run.smithi084.stdout:HOME_URL="https://centos.org/" 2024-03-29T11:08:02.700 INFO:teuthology.orchestra.run.smithi084.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-29T11:08:02.700 INFO:teuthology.orchestra.run.smithi084.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-29T11:08:02.700 INFO:teuthology.orchestra.run.smithi084.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-29T11:08:02.701 INFO:teuthology.lock.ops:Updating smithi084.front.sepia.ceph.com on lock server 2024-03-29T11:08:02.728 DEBUG:teuthology.orchestra.run.smithi179:> uname -m 2024-03-29T11:08:02.745 INFO:teuthology.orchestra.run.smithi179.stdout:x86_64 2024-03-29T11:08:02.745 DEBUG:teuthology.orchestra.run.smithi179:> cat /etc/os-release 2024-03-29T11:08:02.802 INFO:teuthology.orchestra.run.smithi179.stdout:NAME="CentOS Stream" 2024-03-29T11:08:02.802 INFO:teuthology.orchestra.run.smithi179.stdout:VERSION="9" 2024-03-29T11:08:02.802 INFO:teuthology.orchestra.run.smithi179.stdout:ID="centos" 2024-03-29T11:08:02.802 INFO:teuthology.orchestra.run.smithi179.stdout:ID_LIKE="rhel fedora" 2024-03-29T11:08:02.802 INFO:teuthology.orchestra.run.smithi179.stdout:VERSION_ID="9" 2024-03-29T11:08:02.802 INFO:teuthology.orchestra.run.smithi179.stdout:PLATFORM_ID="platform:el9" 2024-03-29T11:08:02.802 INFO:teuthology.orchestra.run.smithi179.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-29T11:08:02.802 INFO:teuthology.orchestra.run.smithi179.stdout:ANSI_COLOR="0;31" 2024-03-29T11:08:02.802 INFO:teuthology.orchestra.run.smithi179.stdout:LOGO="fedora-logo-icon" 2024-03-29T11:08:02.802 INFO:teuthology.orchestra.run.smithi179.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-29T11:08:02.802 INFO:teuthology.orchestra.run.smithi179.stdout:HOME_URL="https://centos.org/" 2024-03-29T11:08:02.803 INFO:teuthology.orchestra.run.smithi179.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-29T11:08:02.803 INFO:teuthology.orchestra.run.smithi179.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-29T11:08:02.803 INFO:teuthology.orchestra.run.smithi179.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-29T11:08:02.803 INFO:teuthology.lock.ops:Updating smithi179.front.sepia.ceph.com on lock server 2024-03-29T11:08:03.069 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-03-29T11:08:03.080 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-03-29T11:08:03.128 INFO:teuthology.task.internal:Checking for old test directory... 2024-03-29T11:08:03.128 DEBUG:teuthology.orchestra.run.smithi084:> test '!' -e /home/ubuntu/cephtest 2024-03-29T11:08:03.131 DEBUG:teuthology.orchestra.run.smithi179:> test '!' -e /home/ubuntu/cephtest 2024-03-29T11:08:03.147 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-03-29T11:08:03.169 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-03-29T11:08:03.169 DEBUG:teuthology.orchestra.run.smithi084:> test -z $(ls -A /var/lib/ceph) 2024-03-29T11:08:03.188 DEBUG:teuthology.orchestra.run.smithi179:> test -z $(ls -A /var/lib/ceph) 2024-03-29T11:08:03.451 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-03-29T11:08:03.505 INFO:teuthology.run_tasks:Running task kernel... 2024-03-29T11:08:03.519 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2024-03-29T11:08:03.520 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}, 'mon.b': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2024-03-29T11:08:03.520 DEBUG:teuthology.orchestra.run.smithi084:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-29T11:08:03.520 DEBUG:teuthology.orchestra.run.smithi179:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-29T11:08:03.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-29T11:08:03.537 DEBUG:teuthology.orchestra.run.smithi084:> uname -r 2024-03-29T11:08:03.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-29T11:08:03.538 DEBUG:teuthology.orchestra.run.smithi179:> uname -r 2024-03-29T11:08:03.594 INFO:teuthology.orchestra.run.smithi084.stdout:5.14.0-361.el9.x86_64 2024-03-29T11:08:03.595 INFO:teuthology.task.kernel:Running kernel on smithi084: 5.14.0-361.el9.x86_64 2024-03-29T11:08:03.595 DEBUG:teuthology.orchestra.run.smithi084:> sudo yum install -y kernel 2024-03-29T11:08:03.596 INFO:teuthology.orchestra.run.smithi179.stdout:5.14.0-361.el9.x86_64 2024-03-29T11:08:03.596 INFO:teuthology.task.kernel:Running kernel on smithi179: 5.14.0-361.el9.x86_64 2024-03-29T11:08:03.596 DEBUG:teuthology.orchestra.run.smithi179:> sudo yum install -y kernel 2024-03-29T11:08:05.726 INFO:teuthology.orchestra.run.smithi084.stdout:CentOS-9 - AppStream 13 MB/s | 6.7 MB 00:00 2024-03-29T11:08:05.780 INFO:teuthology.orchestra.run.smithi179.stdout:CentOS-9 - AppStream 13 MB/s | 6.7 MB 00:00 2024-03-29T11:08:07.157 INFO:teuthology.orchestra.run.smithi179.stdout:CentOS-9 - BaseOS 6.9 MB/s | 2.0 MB 00:00 2024-03-29T11:08:07.351 INFO:teuthology.orchestra.run.smithi084.stdout:CentOS-9 - BaseOS 3.7 MB/s | 2.0 MB 00:00 2024-03-29T11:08:08.250 INFO:teuthology.orchestra.run.smithi084.stdout:CentOS Stream 9 - BaseOS 14 MB/s | 8.0 MB 00:00 2024-03-29T11:08:08.679 INFO:teuthology.orchestra.run.smithi179.stdout:CentOS Stream 9 - BaseOS 6.7 MB/s | 8.0 MB 00:01 2024-03-29T11:08:10.317 INFO:teuthology.orchestra.run.smithi084.stdout:CentOS Stream 9 - AppStream 19 MB/s | 19 MB 00:00 2024-03-29T11:08:14.218 INFO:teuthology.orchestra.run.smithi084.stdout:CentOS Stream 9 - CRB 15 MB/s | 6.1 MB 00:00 2024-03-29T11:08:15.560 INFO:teuthology.orchestra.run.smithi084.stdout:CentOS Stream 9 - Extras packages 70 kB/s | 16 kB 00:00 2024-03-29T11:08:17.122 INFO:teuthology.orchestra.run.smithi084.stdout:Extra Packages for Enterprise Linux 14 MB/s | 21 MB 00:01 2024-03-29T11:08:21.977 INFO:teuthology.orchestra.run.smithi084.stdout:lab-extras 33 kB/s | 1.7 kB 00:00 2024-03-29T11:08:24.250 INFO:teuthology.orchestra.run.smithi084.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-03-29T11:08:24.251 INFO:teuthology.orchestra.run.smithi084.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-03-29T11:08:24.355 INFO:teuthology.orchestra.run.smithi084.stdout:Dependencies resolved. 2024-03-29T11:08:24.366 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2024-03-29T11:08:24.366 INFO:teuthology.orchestra.run.smithi084.stdout: Package Arch Version Repository Size 2024-03-29T11:08:24.367 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2024-03-29T11:08:24.367 INFO:teuthology.orchestra.run.smithi084.stdout:Installing: 2024-03-29T11:08:24.367 INFO:teuthology.orchestra.run.smithi084.stdout: kernel x86_64 5.14.0-434.el9 CentOS-BaseOS 192 k 2024-03-29T11:08:24.367 INFO:teuthology.orchestra.run.smithi084.stdout:Installing dependencies: 2024-03-29T11:08:24.367 INFO:teuthology.orchestra.run.smithi084.stdout: kernel-core x86_64 5.14.0-434.el9 CentOS-BaseOS 15 M 2024-03-29T11:08:24.367 INFO:teuthology.orchestra.run.smithi084.stdout: kernel-modules x86_64 5.14.0-434.el9 CentOS-BaseOS 34 M 2024-03-29T11:08:24.367 INFO:teuthology.orchestra.run.smithi084.stdout: kernel-modules-core x86_64 5.14.0-434.el9 CentOS-BaseOS 28 M 2024-03-29T11:08:24.367 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-03-29T11:08:24.367 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction Summary 2024-03-29T11:08:24.367 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2024-03-29T11:08:24.367 INFO:teuthology.orchestra.run.smithi084.stdout:Install 4 Packages 2024-03-29T11:08:24.367 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-03-29T11:08:24.368 INFO:teuthology.orchestra.run.smithi084.stdout:Total download size: 76 M 2024-03-29T11:08:24.368 INFO:teuthology.orchestra.run.smithi084.stdout:Installed size: 123 M 2024-03-29T11:08:24.368 INFO:teuthology.orchestra.run.smithi084.stdout:Downloading Packages: 2024-03-29T11:08:24.545 INFO:teuthology.orchestra.run.smithi084.stdout:(1/4): kernel-5.14.0-434.el9.x86_64.rpm 1.1 MB/s | 192 kB 00:00 2024-03-29T11:08:25.237 INFO:teuthology.orchestra.run.smithi084.stdout:(2/4): kernel-core-5.14.0-434.el9.x86_64.rpm 17 MB/s | 15 MB 00:00 2024-03-29T11:08:25.737 INFO:teuthology.orchestra.run.smithi084.stdout:(3/4): kernel-modules-5.14.0-434.el9.x86_64.rpm 25 MB/s | 34 MB 00:01 2024-03-29T11:08:26.137 INFO:teuthology.orchestra.run.smithi084.stdout:(4/4): kernel-modules-core-5.14.0-434.el9.x86_6 17 MB/s | 28 MB 00:01 2024-03-29T11:08:26.140 INFO:teuthology.orchestra.run.smithi084.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:08:26.140 INFO:teuthology.orchestra.run.smithi084.stdout:Total 43 MB/s | 76 MB 00:01 2024-03-29T11:08:26.140 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction check 2024-03-29T11:08:26.511 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction check succeeded. 2024-03-29T11:08:26.511 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction test 2024-03-29T11:08:27.343 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction test succeeded. 2024-03-29T11:08:27.343 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction 2024-03-29T11:08:29.614 INFO:teuthology.orchestra.run.smithi084.stdout: Preparing : 1/1 2024-03-29T11:08:30.200 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : kernel-modules-core-5.14.0-434.el9.x86_64 1/4 2024-03-29T11:08:30.337 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : kernel-core-5.14.0-434.el9.x86_64 2/4 2024-03-29T11:08:33.143 INFO:teuthology.orchestra.run.smithi084.stdout: Running scriptlet: kernel-core-5.14.0-434.el9.x86_64 2/4 2024-03-29T11:08:33.282 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : kernel-modules-5.14.0-434.el9.x86_64 3/4 2024-03-29T11:08:35.278 INFO:teuthology.orchestra.run.smithi179.stdout:CentOS Stream 9 - AppStream 757 kB/s | 19 MB 00:25 2024-03-29T11:08:38.030 INFO:teuthology.orchestra.run.smithi084.stdout: Running scriptlet: kernel-modules-5.14.0-434.el9.x86_64 3/4 2024-03-29T11:08:38.138 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : kernel-5.14.0-434.el9.x86_64 4/4 2024-03-29T11:08:40.995 INFO:teuthology.orchestra.run.smithi179.stdout:CentOS Stream 9 - CRB 2.8 MB/s | 6.1 MB 00:02 2024-03-29T11:08:42.479 INFO:teuthology.orchestra.run.smithi179.stdout:CentOS Stream 9 - Extras packages 44 kB/s | 16 kB 00:00 2024-03-29T11:08:42.906 INFO:teuthology.orchestra.run.smithi084.stdout: Running scriptlet: kernel-modules-core-5.14.0-434.el9.x86_64 4/4 2024-03-29T11:08:43.289 INFO:teuthology.orchestra.run.smithi179.stdout:Extra Packages for Enterprise Linux 30 MB/s | 21 MB 00:00 2024-03-29T11:08:48.239 INFO:teuthology.orchestra.run.smithi179.stdout:lab-extras 42 kB/s | 1.7 kB 00:00 2024-03-29T11:08:50.472 INFO:teuthology.orchestra.run.smithi179.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-03-29T11:08:50.472 INFO:teuthology.orchestra.run.smithi179.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-03-29T11:08:50.565 INFO:teuthology.orchestra.run.smithi179.stdout:Dependencies resolved. 2024-03-29T11:08:50.569 INFO:teuthology.orchestra.run.smithi179.stdout:================================================================================ 2024-03-29T11:08:50.569 INFO:teuthology.orchestra.run.smithi179.stdout: Package Arch Version Repository Size 2024-03-29T11:08:50.570 INFO:teuthology.orchestra.run.smithi179.stdout:================================================================================ 2024-03-29T11:08:50.570 INFO:teuthology.orchestra.run.smithi179.stdout:Installing: 2024-03-29T11:08:50.570 INFO:teuthology.orchestra.run.smithi179.stdout: kernel x86_64 5.14.0-434.el9 CentOS-BaseOS 192 k 2024-03-29T11:08:50.570 INFO:teuthology.orchestra.run.smithi179.stdout:Installing dependencies: 2024-03-29T11:08:50.570 INFO:teuthology.orchestra.run.smithi179.stdout: kernel-core x86_64 5.14.0-434.el9 CentOS-BaseOS 15 M 2024-03-29T11:08:50.570 INFO:teuthology.orchestra.run.smithi179.stdout: kernel-modules x86_64 5.14.0-434.el9 CentOS-BaseOS 34 M 2024-03-29T11:08:50.570 INFO:teuthology.orchestra.run.smithi179.stdout: kernel-modules-core x86_64 5.14.0-434.el9 CentOS-BaseOS 28 M 2024-03-29T11:08:50.570 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-29T11:08:50.570 INFO:teuthology.orchestra.run.smithi179.stdout:Transaction Summary 2024-03-29T11:08:50.570 INFO:teuthology.orchestra.run.smithi179.stdout:================================================================================ 2024-03-29T11:08:50.570 INFO:teuthology.orchestra.run.smithi179.stdout:Install 4 Packages 2024-03-29T11:08:50.570 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-29T11:08:50.571 INFO:teuthology.orchestra.run.smithi179.stdout:Total download size: 76 M 2024-03-29T11:08:50.571 INFO:teuthology.orchestra.run.smithi179.stdout:Installed size: 123 M 2024-03-29T11:08:50.571 INFO:teuthology.orchestra.run.smithi179.stdout:Downloading Packages: 2024-03-29T11:08:50.717 INFO:teuthology.orchestra.run.smithi179.stdout:(1/4): kernel-5.14.0-434.el9.x86_64.rpm 1.3 MB/s | 192 kB 00:00 2024-03-29T11:08:51.441 INFO:teuthology.orchestra.run.smithi179.stdout:(2/4): kernel-core-5.14.0-434.el9.x86_64.rpm 17 MB/s | 15 MB 00:00 2024-03-29T11:08:52.075 INFO:teuthology.orchestra.run.smithi179.stdout:(3/4): kernel-modules-5.14.0-434.el9.x86_64.rpm 22 MB/s | 34 MB 00:01 2024-03-29T11:08:52.467 INFO:teuthology.orchestra.run.smithi179.stdout:(4/4): kernel-modules-core-5.14.0-434.el9.x86_6 16 MB/s | 28 MB 00:01 2024-03-29T11:08:52.470 INFO:teuthology.orchestra.run.smithi179.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:08:52.470 INFO:teuthology.orchestra.run.smithi179.stdout:Total 40 MB/s | 76 MB 00:01 2024-03-29T11:08:52.470 INFO:teuthology.orchestra.run.smithi179.stdout:Running transaction check 2024-03-29T11:08:52.845 INFO:teuthology.orchestra.run.smithi179.stdout:Transaction check succeeded. 2024-03-29T11:08:52.845 INFO:teuthology.orchestra.run.smithi179.stdout:Running transaction test 2024-03-29T11:08:53.682 INFO:teuthology.orchestra.run.smithi179.stdout:Transaction test succeeded. 2024-03-29T11:08:53.683 INFO:teuthology.orchestra.run.smithi179.stdout:Running transaction 2024-03-29T11:08:55.970 INFO:teuthology.orchestra.run.smithi179.stdout: Preparing : 1/1 2024-03-29T11:08:56.562 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : kernel-modules-core-5.14.0-434.el9.x86_64 1/4 2024-03-29T11:08:56.656 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : kernel-core-5.14.0-434.el9.x86_64 2/4 2024-03-29T11:08:59.396 INFO:teuthology.orchestra.run.smithi179.stdout: Running scriptlet: kernel-core-5.14.0-434.el9.x86_64 2/4 2024-03-29T11:08:59.538 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : kernel-modules-5.14.0-434.el9.x86_64 3/4 2024-03-29T11:09:04.259 INFO:teuthology.orchestra.run.smithi179.stdout: Running scriptlet: kernel-modules-5.14.0-434.el9.x86_64 3/4 2024-03-29T11:09:04.296 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : kernel-5.14.0-434.el9.x86_64 4/4 2024-03-29T11:09:09.029 INFO:teuthology.orchestra.run.smithi179.stdout: Running scriptlet: kernel-modules-core-5.14.0-434.el9.x86_64 4/4 2024-03-29T11:09:12.459 INFO:teuthology.orchestra.run.smithi084.stdout: Running scriptlet: kernel-core-5.14.0-434.el9.x86_64 4/4 2024-03-29T11:09:12.460 INFO:teuthology.orchestra.run.smithi084.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-29T11:09:12.460 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-03-29T11:09:12.502 INFO:teuthology.orchestra.run.smithi084.stdout: Running scriptlet: kernel-modules-5.14.0-434.el9.x86_64 4/4 2024-03-29T11:09:12.927 INFO:teuthology.orchestra.run.smithi084.stdout: Running scriptlet: kernel-5.14.0-434.el9.x86_64 4/4 2024-03-29T11:09:12.927 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : kernel-5.14.0-434.el9.x86_64 1/4 2024-03-29T11:09:12.927 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : kernel-core-5.14.0-434.el9.x86_64 2/4 2024-03-29T11:09:12.927 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : kernel-modules-5.14.0-434.el9.x86_64 3/4 2024-03-29T11:09:13.337 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : kernel-modules-core-5.14.0-434.el9.x86_64 4/4 2024-03-29T11:09:13.337 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-03-29T11:09:13.337 INFO:teuthology.orchestra.run.smithi084.stdout:Installed: 2024-03-29T11:09:13.338 INFO:teuthology.orchestra.run.smithi084.stdout: kernel-5.14.0-434.el9.x86_64 2024-03-29T11:09:13.338 INFO:teuthology.orchestra.run.smithi084.stdout: kernel-core-5.14.0-434.el9.x86_64 2024-03-29T11:09:13.338 INFO:teuthology.orchestra.run.smithi084.stdout: kernel-modules-5.14.0-434.el9.x86_64 2024-03-29T11:09:13.338 INFO:teuthology.orchestra.run.smithi084.stdout: kernel-modules-core-5.14.0-434.el9.x86_64 2024-03-29T11:09:13.338 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-03-29T11:09:13.338 INFO:teuthology.orchestra.run.smithi084.stdout:Complete! 2024-03-29T11:09:13.678 DEBUG:teuthology.orchestra.run.smithi084:> rpm -q kernel | sort -rV | head -n 1 2024-03-29T11:09:13.817 INFO:teuthology.orchestra.run.smithi084.stdout:kernel-5.14.0-434.el9.x86_64 2024-03-29T11:09:13.817 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-434.el9.x86_64 2024-03-29T11:09:13.817 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-434.el9.x86_64 2024-03-29T11:09:13.817 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-29T11:09:13.817 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-29T11:09:13.817 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-03-29T11:09:13.818 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi084.front.sepia.ceph.com, path=None, version=distro) 2024-03-29T11:09:13.818 DEBUG:teuthology.orchestra.run.smithi084:> rpm -q kernel | sort -rV | head -n 1 2024-03-29T11:09:13.897 INFO:teuthology.orchestra.run.smithi084.stdout:kernel-5.14.0-434.el9.x86_64 2024-03-29T11:09:13.898 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-434.el9.x86_64 2024-03-29T11:09:13.898 DEBUG:teuthology.orchestra.run.smithi084:> sudo rpm -qi grub2-tools 2024-03-29T11:09:13.940 INFO:teuthology.orchestra.run.smithi084.stdout:Name : grub2-tools 2024-03-29T11:09:13.940 INFO:teuthology.orchestra.run.smithi084.stdout:Epoch : 1 2024-03-29T11:09:13.941 INFO:teuthology.orchestra.run.smithi084.stdout:Version : 2.06 2024-03-29T11:09:13.941 INFO:teuthology.orchestra.run.smithi084.stdout:Release : 68.el9 2024-03-29T11:09:13.941 INFO:teuthology.orchestra.run.smithi084.stdout:Architecture: x86_64 2024-03-29T11:09:13.941 INFO:teuthology.orchestra.run.smithi084.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-29T11:09:13.941 INFO:teuthology.orchestra.run.smithi084.stdout:Group : Unspecified 2024-03-29T11:09:13.941 INFO:teuthology.orchestra.run.smithi084.stdout:Size : 8264254 2024-03-29T11:09:13.941 INFO:teuthology.orchestra.run.smithi084.stdout:License : GPLv3+ 2024-03-29T11:09:13.941 INFO:teuthology.orchestra.run.smithi084.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-29T11:09:13.941 INFO:teuthology.orchestra.run.smithi084.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-29T11:09:13.941 INFO:teuthology.orchestra.run.smithi084.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-29T11:09:13.941 INFO:teuthology.orchestra.run.smithi084.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-29T11:09:13.941 INFO:teuthology.orchestra.run.smithi084.stdout:Packager : builder@centos.org 2024-03-29T11:09:13.941 INFO:teuthology.orchestra.run.smithi084.stdout:Vendor : CentOS 2024-03-29T11:09:13.941 INFO:teuthology.orchestra.run.smithi084.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-29T11:09:13.942 INFO:teuthology.orchestra.run.smithi084.stdout:Summary : Support tools for GRUB. 2024-03-29T11:09:13.942 INFO:teuthology.orchestra.run.smithi084.stdout:Description : 2024-03-29T11:09:13.942 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-03-29T11:09:13.942 INFO:teuthology.orchestra.run.smithi084.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-29T11:09:13.942 INFO:teuthology.orchestra.run.smithi084.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-29T11:09:13.942 INFO:teuthology.orchestra.run.smithi084.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-29T11:09:13.942 INFO:teuthology.orchestra.run.smithi084.stdout:hardware devices. 2024-03-29T11:09:13.942 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-03-29T11:09:13.942 INFO:teuthology.orchestra.run.smithi084.stdout:This subpackage provides tools for support of all platforms. 2024-03-29T11:09:13.943 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-29T11:09:13.943 INFO:teuthology.task.kernel:Updating grub on smithi084 to boot 5.14.0-434.el9.x86_64 2024-03-29T11:09:13.943 DEBUG:teuthology.orchestra.run.smithi084:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-29T11:09:14.593 INFO:teuthology.orchestra.run.smithi084.stderr:Generating grub configuration file ... 2024-03-29T11:09:15.942 INFO:teuthology.orchestra.run.smithi084.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-29T11:09:15.985 INFO:teuthology.orchestra.run.smithi084.stderr:done 2024-03-29T11:09:15.987 DEBUG:teuthology.orchestra.run.smithi084:> mktemp 2024-03-29T11:09:16.003 INFO:teuthology.orchestra.run.smithi084.stdout:/tmp/tmp.JQClPnYAkt 2024-03-29T11:09:16.004 DEBUG:teuthology.orchestra.run.smithi084:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.JQClPnYAkt 2024-03-29T11:09:16.081 DEBUG:teuthology.orchestra.run.smithi084:> sudo chmod 0666 /tmp/tmp.JQClPnYAkt 2024-03-29T11:09:16.213 DEBUG:teuthology.orchestra.remote:smithi084:/tmp/tmp.JQClPnYAkt is 6KB 2024-03-29T11:09:16.261 DEBUG:teuthology.orchestra.run.smithi084:> rm -fr /tmp/tmp.JQClPnYAkt 2024-03-29T11:09:16.276 DEBUG:teuthology.orchestra.run.smithi084:> sudo /bin/ls /boot/loader/entries || true 2024-03-29T11:09:16.340 INFO:teuthology.orchestra.run.smithi084.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-29T11:09:16.341 INFO:teuthology.orchestra.run.smithi084.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-29T11:09:16.341 INFO:teuthology.orchestra.run.smithi084.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-29T11:09:16.341 INFO:teuthology.orchestra.run.smithi084.stdout:15949e3c88704300882482eddfd7baae-5.14.0-434.el9.x86_64.conf 2024-03-29T11:09:16.342 DEBUG:teuthology.orchestra.run.smithi084:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-434.el9.x86_64 2024-03-29T11:09:16.502 DEBUG:teuthology.orchestra.run.smithi084:> sudo shutdown -r now 2024-03-29T11:09:38.537 INFO:teuthology.orchestra.run.smithi179.stdout: Running scriptlet: kernel-core-5.14.0-434.el9.x86_64 4/4 2024-03-29T11:09:38.537 INFO:teuthology.orchestra.run.smithi179.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-29T11:09:38.537 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-29T11:09:38.581 INFO:teuthology.orchestra.run.smithi179.stdout: Running scriptlet: kernel-modules-5.14.0-434.el9.x86_64 4/4 2024-03-29T11:09:39.014 INFO:teuthology.orchestra.run.smithi179.stdout: Running scriptlet: kernel-5.14.0-434.el9.x86_64 4/4 2024-03-29T11:09:39.014 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : kernel-5.14.0-434.el9.x86_64 1/4 2024-03-29T11:09:39.014 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : kernel-core-5.14.0-434.el9.x86_64 2/4 2024-03-29T11:09:39.014 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : kernel-modules-5.14.0-434.el9.x86_64 3/4 2024-03-29T11:09:39.429 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : kernel-modules-core-5.14.0-434.el9.x86_64 4/4 2024-03-29T11:09:39.429 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-29T11:09:39.429 INFO:teuthology.orchestra.run.smithi179.stdout:Installed: 2024-03-29T11:09:39.429 INFO:teuthology.orchestra.run.smithi179.stdout: kernel-5.14.0-434.el9.x86_64 2024-03-29T11:09:39.429 INFO:teuthology.orchestra.run.smithi179.stdout: kernel-core-5.14.0-434.el9.x86_64 2024-03-29T11:09:39.429 INFO:teuthology.orchestra.run.smithi179.stdout: kernel-modules-5.14.0-434.el9.x86_64 2024-03-29T11:09:39.429 INFO:teuthology.orchestra.run.smithi179.stdout: kernel-modules-core-5.14.0-434.el9.x86_64 2024-03-29T11:09:39.429 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-29T11:09:39.429 INFO:teuthology.orchestra.run.smithi179.stdout:Complete! 2024-03-29T11:09:39.799 DEBUG:teuthology.orchestra.run.smithi179:> rpm -q kernel | sort -rV | head -n 1 2024-03-29T11:09:39.904 INFO:teuthology.orchestra.run.smithi179.stdout:kernel-5.14.0-434.el9.x86_64 2024-03-29T11:09:39.905 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-434.el9.x86_64 2024-03-29T11:09:39.905 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-434.el9.x86_64 2024-03-29T11:09:39.906 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-29T11:09:39.906 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-29T11:09:39.906 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2024-03-29T11:09:39.906 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi179.front.sepia.ceph.com, path=None, version=distro) 2024-03-29T11:09:39.906 DEBUG:teuthology.orchestra.run.smithi179:> rpm -q kernel | sort -rV | head -n 1 2024-03-29T11:09:39.985 INFO:teuthology.orchestra.run.smithi179.stdout:kernel-5.14.0-434.el9.x86_64 2024-03-29T11:09:39.985 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-434.el9.x86_64 2024-03-29T11:09:39.985 DEBUG:teuthology.orchestra.run.smithi179:> sudo rpm -qi grub2-tools 2024-03-29T11:09:40.028 INFO:teuthology.orchestra.run.smithi179.stdout:Name : grub2-tools 2024-03-29T11:09:40.028 INFO:teuthology.orchestra.run.smithi179.stdout:Epoch : 1 2024-03-29T11:09:40.028 INFO:teuthology.orchestra.run.smithi179.stdout:Version : 2.06 2024-03-29T11:09:40.028 INFO:teuthology.orchestra.run.smithi179.stdout:Release : 68.el9 2024-03-29T11:09:40.028 INFO:teuthology.orchestra.run.smithi179.stdout:Architecture: x86_64 2024-03-29T11:09:40.028 INFO:teuthology.orchestra.run.smithi179.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-29T11:09:40.029 INFO:teuthology.orchestra.run.smithi179.stdout:Group : Unspecified 2024-03-29T11:09:40.029 INFO:teuthology.orchestra.run.smithi179.stdout:Size : 8264254 2024-03-29T11:09:40.029 INFO:teuthology.orchestra.run.smithi179.stdout:License : GPLv3+ 2024-03-29T11:09:40.029 INFO:teuthology.orchestra.run.smithi179.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-29T11:09:40.029 INFO:teuthology.orchestra.run.smithi179.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-29T11:09:40.029 INFO:teuthology.orchestra.run.smithi179.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-29T11:09:40.029 INFO:teuthology.orchestra.run.smithi179.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-29T11:09:40.029 INFO:teuthology.orchestra.run.smithi179.stdout:Packager : builder@centos.org 2024-03-29T11:09:40.029 INFO:teuthology.orchestra.run.smithi179.stdout:Vendor : CentOS 2024-03-29T11:09:40.029 INFO:teuthology.orchestra.run.smithi179.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-29T11:09:40.029 INFO:teuthology.orchestra.run.smithi179.stdout:Summary : Support tools for GRUB. 2024-03-29T11:09:40.029 INFO:teuthology.orchestra.run.smithi179.stdout:Description : 2024-03-29T11:09:40.029 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-29T11:09:40.029 INFO:teuthology.orchestra.run.smithi179.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-29T11:09:40.029 INFO:teuthology.orchestra.run.smithi179.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-29T11:09:40.030 INFO:teuthology.orchestra.run.smithi179.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-29T11:09:40.030 INFO:teuthology.orchestra.run.smithi179.stdout:hardware devices. 2024-03-29T11:09:40.030 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-29T11:09:40.030 INFO:teuthology.orchestra.run.smithi179.stdout:This subpackage provides tools for support of all platforms. 2024-03-29T11:09:40.031 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-29T11:09:40.031 INFO:teuthology.task.kernel:Updating grub on smithi179 to boot 5.14.0-434.el9.x86_64 2024-03-29T11:09:40.031 DEBUG:teuthology.orchestra.run.smithi179:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-29T11:09:40.721 INFO:teuthology.orchestra.run.smithi179.stderr:Generating grub configuration file ... 2024-03-29T11:09:42.104 INFO:teuthology.orchestra.run.smithi179.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-29T11:09:42.146 INFO:teuthology.orchestra.run.smithi179.stderr:done 2024-03-29T11:09:42.148 DEBUG:teuthology.orchestra.run.smithi179:> mktemp 2024-03-29T11:09:42.163 INFO:teuthology.orchestra.run.smithi179.stdout:/tmp/tmp.R4uAIlWY9x 2024-03-29T11:09:42.164 DEBUG:teuthology.orchestra.run.smithi179:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.R4uAIlWY9x 2024-03-29T11:09:42.232 DEBUG:teuthology.orchestra.run.smithi179:> sudo chmod 0666 /tmp/tmp.R4uAIlWY9x 2024-03-29T11:09:42.376 DEBUG:teuthology.orchestra.remote:smithi179:/tmp/tmp.R4uAIlWY9x is 6KB 2024-03-29T11:09:42.424 DEBUG:teuthology.orchestra.run.smithi179:> rm -fr /tmp/tmp.R4uAIlWY9x 2024-03-29T11:09:42.439 DEBUG:teuthology.orchestra.run.smithi179:> sudo /bin/ls /boot/loader/entries || true 2024-03-29T11:09:42.504 INFO:teuthology.orchestra.run.smithi179.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-29T11:09:42.504 INFO:teuthology.orchestra.run.smithi179.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-29T11:09:42.504 INFO:teuthology.orchestra.run.smithi179.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-29T11:09:42.504 INFO:teuthology.orchestra.run.smithi179.stdout:15949e3c88704300882482eddfd7baae-5.14.0-434.el9.x86_64.conf 2024-03-29T11:09:42.505 DEBUG:teuthology.orchestra.run.smithi179:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-434.el9.x86_64 2024-03-29T11:09:42.644 DEBUG:teuthology.orchestra.run.smithi179:> sudo shutdown -r now 2024-03-29T11:09:46.506 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-03-29T11:09:46.506 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi084.front.sepia.ceph.com' 2024-03-29T11:09:46.507 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi084.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T11:10:04.831 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.84 2024-03-29T11:10:12.650 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2024-03-29T11:10:12.650 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi179.front.sepia.ceph.com' 2024-03-29T11:10:12.651 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi179.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T11:10:13.833 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi084.front.sepia.ceph.com' 2024-03-29T11:10:13.834 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi084.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T11:10:16.894 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.84 2024-03-29T11:10:28.907 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi084.front.sepia.ceph.com' 2024-03-29T11:10:28.907 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi084.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T11:10:31.201 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.179 2024-03-29T11:10:40.210 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi179.front.sepia.ceph.com' 2024-03-29T11:10:40.211 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi179.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T11:10:43.262 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.179 2024-03-29T11:10:47.331 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.84 2024-03-29T11:10:55.270 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi179.front.sepia.ceph.com' 2024-03-29T11:10:55.271 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi179.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T11:11:02.338 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi084.front.sepia.ceph.com' 2024-03-29T11:11:02.339 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi084.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T11:11:02.733 DEBUG:teuthology.orchestra.run.smithi084:> true 2024-03-29T11:11:03.154 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi084.front.sepia.ceph.com' 2024-03-29T11:11:03.155 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.14.0-434.el9.x86_64"... 2024-03-29T11:11:03.155 DEBUG:teuthology.orchestra.run.smithi084:> uname -r 2024-03-29T11:11:03.169 INFO:teuthology.orchestra.run.smithi084.stdout:5.14.0-434.el9.x86_64 2024-03-29T11:11:03.169 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-434.el9.x86_64 vs 5.14.0-434.el9.x86_64 2024-03-29T11:11:03.169 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-29T11:11:03.170 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-29T11:11:04.170 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-03-29T11:11:04.170 DEBUG:teuthology.orchestra.run.smithi084:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-29T11:11:04.351 INFO:teuthology.orchestra.run.smithi084.stdout:ttyS1 2024-03-29T11:11:04.410 DEBUG:teuthology.parallel:result is None 2024-03-29T11:11:27.081 DEBUG:teuthology.orchestra.run.smithi179:> true 2024-03-29T11:11:27.546 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi179.front.sepia.ceph.com' 2024-03-29T11:11:27.546 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "5.14.0-434.el9.x86_64"... 2024-03-29T11:11:27.546 DEBUG:teuthology.orchestra.run.smithi179:> uname -r 2024-03-29T11:11:27.560 INFO:teuthology.orchestra.run.smithi179.stdout:5.14.0-434.el9.x86_64 2024-03-29T11:11:27.560 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-434.el9.x86_64 vs 5.14.0-434.el9.x86_64 2024-03-29T11:11:27.560 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-29T11:11:27.560 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-29T11:11:28.561 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2024-03-29T11:11:28.561 DEBUG:teuthology.orchestra.run.smithi179:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-29T11:11:28.628 INFO:teuthology.orchestra.run.smithi179.stdout:ttyS1 2024-03-29T11:11:28.657 DEBUG:teuthology.parallel:result is None 2024-03-29T11:11:28.657 INFO:teuthology.run_tasks:Running task internal.base... 2024-03-29T11:11:28.666 INFO:teuthology.task.internal:Creating test directory... 2024-03-29T11:11:28.666 DEBUG:teuthology.orchestra.run.smithi084:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-29T11:11:28.669 DEBUG:teuthology.orchestra.run.smithi179:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-29T11:11:28.685 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-03-29T11:11:28.700 INFO:teuthology.run_tasks:Running task internal.archive... 2024-03-29T11:11:28.745 INFO:teuthology.task.internal:Creating archive directory... 2024-03-29T11:11:28.746 DEBUG:teuthology.orchestra.run.smithi084:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-29T11:11:28.748 DEBUG:teuthology.orchestra.run.smithi179:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-29T11:11:28.824 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-03-29T11:11:28.831 INFO:teuthology.task.internal:Enabling coredump saving... 2024-03-29T11:11:28.831 DEBUG:teuthology.orchestra.run.smithi084:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-29T11:11:28.834 DEBUG:teuthology.orchestra.run.smithi179:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-29T11:11:28.872 INFO:teuthology.orchestra.run.smithi084.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-29T11:11:28.878 INFO:teuthology.orchestra.run.smithi179.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-29T11:11:28.886 INFO:teuthology.orchestra.run.smithi084.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-29T11:11:28.891 INFO:teuthology.orchestra.run.smithi179.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-29T11:11:28.893 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-03-29T11:11:28.900 INFO:teuthology.task.internal:Configuring sudo... 2024-03-29T11:11:28.900 DEBUG:teuthology.orchestra.run.smithi084:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-29T11:11:28.931 DEBUG:teuthology.orchestra.run.smithi179:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-29T11:11:28.960 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-03-29T11:11:28.988 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-03-29T11:11:28.988 DEBUG:teuthology.orchestra.run.smithi084:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-29T11:11:29.002 DEBUG:teuthology.orchestra.run.smithi179:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-29T11:11:29.016 DEBUG:teuthology.orchestra.run.smithi084:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-29T11:11:29.097 DEBUG:teuthology.orchestra.run.smithi084:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-29T11:11:29.156 DEBUG:teuthology.orchestra.run.smithi084:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-29T11:11:29.235 DEBUG:teuthology.orchestra.run.smithi084:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-29T11:11:29.283 DEBUG:teuthology.orchestra.run.smithi084:> set -ex 2024-03-29T11:11:29.283 DEBUG:teuthology.orchestra.run.smithi084:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-29T11:11:29.350 DEBUG:teuthology.orchestra.run.smithi179:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-29T11:11:29.391 DEBUG:teuthology.orchestra.run.smithi179:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-29T11:11:29.433 DEBUG:teuthology.orchestra.run.smithi179:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-29T11:11:29.507 DEBUG:teuthology.orchestra.run.smithi179:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-29T11:11:29.549 DEBUG:teuthology.orchestra.run.smithi179:> set -ex 2024-03-29T11:11:29.549 DEBUG:teuthology.orchestra.run.smithi179:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-29T11:11:29.613 DEBUG:teuthology.orchestra.run.smithi084:> sudo service rsyslog restart 2024-03-29T11:11:29.615 DEBUG:teuthology.orchestra.run.smithi179:> sudo service rsyslog restart 2024-03-29T11:11:29.673 INFO:teuthology.orchestra.run.smithi084.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-29T11:11:29.729 INFO:teuthology.orchestra.run.smithi179.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-29T11:11:30.173 INFO:teuthology.run_tasks:Running task internal.timer... 2024-03-29T11:11:30.185 INFO:teuthology.task.internal:Starting timer... 2024-03-29T11:11:30.185 INFO:teuthology.run_tasks:Running task pcp... 2024-03-29T11:11:30.221 INFO:teuthology.run_tasks:Running task selinux... 2024-03-29T11:11:30.269 DEBUG:teuthology.orchestra.run.smithi084:> sudo service auditd rotate 2024-03-29T11:11:30.318 INFO:teuthology.orchestra.run.smithi084.stdout:Rotating logs: 2024-03-29T11:11:30.319 DEBUG:teuthology.orchestra.run.smithi179:> sudo service auditd rotate 2024-03-29T11:11:30.405 INFO:teuthology.orchestra.run.smithi179.stdout:Rotating logs: 2024-03-29T11:11:30.407 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-03-29T11:11:30.407 DEBUG:teuthology.orchestra.run.smithi084:> /usr/sbin/getenforce 2024-03-29T11:11:30.425 INFO:teuthology.orchestra.run.smithi084.stdout:Permissive 2024-03-29T11:11:30.425 DEBUG:teuthology.orchestra.run.smithi179:> /usr/sbin/getenforce 2024-03-29T11:11:30.476 INFO:teuthology.orchestra.run.smithi179.stdout:Permissive 2024-03-29T11:11:30.476 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi084.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi179.front.sepia.ceph.com': 'permissive'} 2024-03-29T11:11:30.476 DEBUG:teuthology.orchestra.run.smithi084:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-29T11:11:30.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-29T11:11:30.505 DEBUG:teuthology.orchestra.run.smithi179:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-29T11:11:30.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-29T11:11:30.533 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-03-29T11:11:30.533 DEBUG:teuthology.orchestra.run.smithi084:> sudo /usr/sbin/setenforce permissive 2024-03-29T11:11:30.577 DEBUG:teuthology.orchestra.run.smithi179:> sudo /usr/sbin/setenforce permissive 2024-03-29T11:11:30.736 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-03-29T11:11:30.748 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-03-29T11:11:30.764 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-03-29T11:11:30.766 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi084.front.sepia.ceph.com,smithi179.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-03-29T11:15:48.934 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi084.front.sepia.ceph.com'), Remote(name='ubuntu@smithi179.front.sepia.ceph.com')] 2024-03-29T11:15:48.935 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi084.front.sepia.ceph.com' 2024-03-29T11:15:48.937 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi084.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T11:15:49.041 DEBUG:teuthology.orchestra.run.smithi084:> true 2024-03-29T11:15:49.121 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi084.front.sepia.ceph.com' 2024-03-29T11:15:49.122 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi179.front.sepia.ceph.com' 2024-03-29T11:15:49.122 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi179.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T11:15:49.209 DEBUG:teuthology.orchestra.run.smithi179:> true 2024-03-29T11:15:49.294 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi179.front.sepia.ceph.com' 2024-03-29T11:15:49.294 INFO:teuthology.run_tasks:Running task clock... 2024-03-29T11:15:49.312 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-03-29T11:15:49.312 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-29T11:15:49.313 DEBUG:teuthology.orchestra.run.smithi084:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-29T11:15:49.315 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-29T11:15:49.315 DEBUG:teuthology.orchestra.run.smithi179:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-29T11:15:49.344 INFO:teuthology.orchestra.run.smithi084.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-29T11:15:49.360 INFO:teuthology.orchestra.run.smithi084.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-29T11:15:49.366 INFO:teuthology.orchestra.run.smithi179.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-29T11:15:49.383 INFO:teuthology.orchestra.run.smithi179.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-29T11:15:49.389 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: ntpd: command not found 2024-03-29T11:15:49.402 INFO:teuthology.orchestra.run.smithi084.stdout:506 Cannot talk to daemon 2024-03-29T11:15:49.412 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: ntpd: command not found 2024-03-29T11:15:49.419 INFO:teuthology.orchestra.run.smithi084.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-29T11:15:49.426 INFO:teuthology.orchestra.run.smithi179.stdout:506 Cannot talk to daemon 2024-03-29T11:15:49.435 INFO:teuthology.orchestra.run.smithi084.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-29T11:15:49.442 INFO:teuthology.orchestra.run.smithi179.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-29T11:15:49.458 INFO:teuthology.orchestra.run.smithi179.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-29T11:15:49.491 INFO:teuthology.orchestra.run.smithi084.stderr:bash: line 1: ntpq: command not found 2024-03-29T11:15:49.495 INFO:teuthology.orchestra.run.smithi084.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-29T11:15:49.495 INFO:teuthology.orchestra.run.smithi084.stdout:=============================================================================== 2024-03-29T11:15:49.495 INFO:teuthology.orchestra.run.smithi084.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T11:15:49.495 INFO:teuthology.orchestra.run.smithi084.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T11:15:49.495 INFO:teuthology.orchestra.run.smithi084.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T11:15:49.495 INFO:teuthology.orchestra.run.smithi084.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T11:15:49.515 INFO:teuthology.orchestra.run.smithi179.stderr:bash: line 1: ntpq: command not found 2024-03-29T11:15:49.519 INFO:teuthology.orchestra.run.smithi179.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-29T11:15:49.519 INFO:teuthology.orchestra.run.smithi179.stdout:=============================================================================== 2024-03-29T11:15:49.519 INFO:teuthology.orchestra.run.smithi179.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T11:15:49.519 INFO:teuthology.orchestra.run.smithi179.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T11:15:49.519 INFO:teuthology.orchestra.run.smithi179.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T11:15:49.519 INFO:teuthology.orchestra.run.smithi179.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T11:15:49.519 INFO:teuthology.run_tasks:Running task install... 2024-03-29T11:15:49.530 DEBUG:teuthology.task.install:project ceph 2024-03-29T11:15:49.530 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2'}} 2024-03-29T11:15:49.531 DEBUG:teuthology.task.install:config {'flavor': 'default', 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2'} 2024-03-29T11:15:49.531 INFO:teuthology.task.install:Using flavor: default 2024-03-29T11:15:49.538 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-03-29T11:15:49.538 INFO:teuthology.task.install:extra packages: [] 2024-03-29T11:15:49.539 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2', 'tag': None, 'wait_for_package': False} 2024-03-29T11:15:49.539 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=5dcb190da270440ef141d088eb27a9fdb8afc5f2 2024-03-29T11:15:49.541 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2', 'tag': None, 'wait_for_package': False} 2024-03-29T11:15:49.542 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=5dcb190da270440ef141d088eb27a9fdb8afc5f2 2024-03-29T11:15:49.689 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/ 2024-03-29T11:15:49.689 INFO:teuthology.task.install.rpm:Package version is 19.0.0-1439.g5dcb190d 2024-03-29T11:15:49.757 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/ 2024-03-29T11:15:49.757 INFO:teuthology.task.install.rpm:Package version is 19.0.0-1439.g5dcb190d 2024-03-29T11:15:49.956 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-29T11:15:49.956 DEBUG:teuthology.orchestra.run.smithi084:> set -ex 2024-03-29T11:15:49.956 DEBUG:teuthology.orchestra.run.smithi084:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-29T11:15:49.989 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-03-29T11:15:49.989 DEBUG:teuthology.orchestra.run.smithi084:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/5dcb190da270440ef141d088eb27a9fdb8afc5f2/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-29T11:15:49.989 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-29T11:15:49.989 DEBUG:teuthology.orchestra.run.smithi179:> set -ex 2024-03-29T11:15:49.990 DEBUG:teuthology.orchestra.run.smithi179:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-29T11:15:50.018 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-03-29T11:15:50.018 DEBUG:teuthology.orchestra.run.smithi179:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/5dcb190da270440ef141d088eb27a9fdb8afc5f2/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-29T11:15:50.057 DEBUG:teuthology.orchestra.run.smithi084:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-29T11:15:50.087 DEBUG:teuthology.orchestra.run.smithi179:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-29T11:15:50.137 DEBUG:teuthology.orchestra.run.smithi084:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-29T11:15:50.167 DEBUG:teuthology.orchestra.run.smithi179:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-29T11:15:50.203 INFO:teuthology.orchestra.run.smithi084.stdout:check_obsoletes = 1 2024-03-29T11:15:50.205 DEBUG:teuthology.orchestra.run.smithi084:> sudo yum clean all 2024-03-29T11:15:50.234 INFO:teuthology.orchestra.run.smithi179.stdout:check_obsoletes = 1 2024-03-29T11:15:50.235 DEBUG:teuthology.orchestra.run.smithi179:> sudo yum clean all 2024-03-29T11:15:50.446 INFO:teuthology.orchestra.run.smithi084.stdout:74 files removed 2024-03-29T11:15:50.473 DEBUG:teuthology.orchestra.run.smithi084:> sudo yum -y install ceph-radosgw 2024-03-29T11:15:50.476 INFO:teuthology.orchestra.run.smithi179.stdout:74 files removed 2024-03-29T11:15:50.503 DEBUG:teuthology.orchestra.run.smithi179:> sudo yum -y install ceph-radosgw 2024-03-29T11:15:51.264 INFO:teuthology.orchestra.run.smithi179.stdout:ceph packages for x86_64 178 kB/s | 82 kB 00:00 2024-03-29T11:15:51.310 INFO:teuthology.orchestra.run.smithi084.stdout:ceph packages for x86_64 152 kB/s | 82 kB 00:00 2024-03-29T11:15:51.757 INFO:teuthology.orchestra.run.smithi179.stdout:ceph noarch packages 23 kB/s | 10 kB 00:00 2024-03-29T11:15:51.760 INFO:teuthology.orchestra.run.smithi084.stdout:ceph noarch packages 25 kB/s | 10 kB 00:00 2024-03-29T11:15:52.311 INFO:teuthology.orchestra.run.smithi084.stdout:ceph source packages 3.7 kB/s | 1.9 kB 00:00 2024-03-29T11:15:52.382 INFO:teuthology.orchestra.run.smithi179.stdout:ceph source packages 3.2 kB/s | 1.9 kB 00:00 2024-03-29T11:15:52.926 INFO:teuthology.orchestra.run.smithi179.stdout:CentOS-9 - AppStream 13 MB/s | 6.7 MB 00:00 2024-03-29T11:15:53.038 INFO:teuthology.orchestra.run.smithi084.stdout:CentOS-9 - AppStream 9.7 MB/s | 6.7 MB 00:00 2024-03-29T11:15:54.535 INFO:teuthology.orchestra.run.smithi179.stdout:CentOS-9 - BaseOS 4.1 MB/s | 2.0 MB 00:00 2024-03-29T11:15:54.555 INFO:teuthology.orchestra.run.smithi084.stdout:CentOS-9 - BaseOS 5.2 MB/s | 2.0 MB 00:00 2024-03-29T11:15:55.693 INFO:teuthology.orchestra.run.smithi179.stdout:CentOS Stream 9 - BaseOS 9.7 MB/s | 8.0 MB 00:00 2024-03-29T11:15:56.838 INFO:teuthology.orchestra.run.smithi084.stdout:CentOS Stream 9 - BaseOS 4.1 MB/s | 8.0 MB 00:01 2024-03-29T11:15:57.879 INFO:teuthology.orchestra.run.smithi179.stdout:CentOS Stream 9 - AppStream 17 MB/s | 19 MB 00:01 2024-03-29T11:16:00.547 INFO:teuthology.orchestra.run.smithi084.stdout:CentOS Stream 9 - AppStream 7.2 MB/s | 19 MB 00:02 2024-03-29T11:16:05.055 INFO:teuthology.orchestra.run.smithi084.stdout:CentOS Stream 9 - CRB 5.9 MB/s | 6.1 MB 00:01 2024-03-29T11:16:06.682 INFO:teuthology.orchestra.run.smithi084.stdout:CentOS Stream 9 - Extras packages 33 kB/s | 16 kB 00:00 2024-03-29T11:16:08.093 INFO:teuthology.orchestra.run.smithi084.stdout:Extra Packages for Enterprise Linux 16 MB/s | 21 MB 00:01 2024-03-29T11:16:09.818 INFO:teuthology.orchestra.run.smithi179.stdout:CentOS Stream 9 - CRB 743 kB/s | 6.1 MB 00:08 2024-03-29T11:16:11.161 INFO:teuthology.orchestra.run.smithi179.stdout:CentOS Stream 9 - Extras packages 89 kB/s | 16 kB 00:00 2024-03-29T11:16:12.738 INFO:teuthology.orchestra.run.smithi179.stdout:Extra Packages for Enterprise Linux 14 MB/s | 21 MB 00:01 2024-03-29T11:16:12.973 INFO:teuthology.orchestra.run.smithi084.stdout:lab-extras 45 kB/s | 1.7 kB 00:00 2024-03-29T11:16:15.641 INFO:teuthology.orchestra.run.smithi084.stdout:Dependencies resolved. 2024-03-29T11:16:15.660 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2024-03-29T11:16:15.660 INFO:teuthology.orchestra.run.smithi084.stdout: Package Arch Version Repository Size 2024-03-29T11:16:15.660 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2024-03-29T11:16:15.660 INFO:teuthology.orchestra.run.smithi084.stdout:Installing: 2024-03-29T11:16:15.660 INFO:teuthology.orchestra.run.smithi084.stdout: ceph-radosgw x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 9.7 M 2024-03-29T11:16:15.661 INFO:teuthology.orchestra.run.smithi084.stdout:Upgrading: 2024-03-29T11:16:15.661 INFO:teuthology.orchestra.run.smithi084.stdout: librados2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.4 M 2024-03-29T11:16:15.661 INFO:teuthology.orchestra.run.smithi084.stdout: librbd1 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.1 M 2024-03-29T11:16:15.661 INFO:teuthology.orchestra.run.smithi084.stdout:Installing dependencies: 2024-03-29T11:16:15.661 INFO:teuthology.orchestra.run.smithi084.stdout: boost-program-options 2024-03-29T11:16:15.661 INFO:teuthology.orchestra.run.smithi084.stdout: x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-29T11:16:15.661 INFO:teuthology.orchestra.run.smithi084.stdout: ceph-base x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.3 M 2024-03-29T11:16:15.661 INFO:teuthology.orchestra.run.smithi084.stdout: ceph-common x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 20 M 2024-03-29T11:16:15.661 INFO:teuthology.orchestra.run.smithi084.stdout: ceph-selinux x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 25 k 2024-03-29T11:16:15.661 INFO:teuthology.orchestra.run.smithi084.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-29T11:16:15.661 INFO:teuthology.orchestra.run.smithi084.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-29T11:16:15.661 INFO:teuthology.orchestra.run.smithi084.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-29T11:16:15.661 INFO:teuthology.orchestra.run.smithi084.stdout: libcephfs2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 742 k 2024-03-29T11:16:15.661 INFO:teuthology.orchestra.run.smithi084.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-29T11:16:15.662 INFO:teuthology.orchestra.run.smithi084.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-29T11:16:15.662 INFO:teuthology.orchestra.run.smithi084.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-29T11:16:15.662 INFO:teuthology.orchestra.run.smithi084.stdout: libradosstriper1 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 498 k 2024-03-29T11:16:15.662 INFO:teuthology.orchestra.run.smithi084.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-29T11:16:15.662 INFO:teuthology.orchestra.run.smithi084.stdout: librgw2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.1 M 2024-03-29T11:16:15.662 INFO:teuthology.orchestra.run.smithi084.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-29T11:16:15.662 INFO:teuthology.orchestra.run.smithi084.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-29T11:16:15.662 INFO:teuthology.orchestra.run.smithi084.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-29T11:16:15.662 INFO:teuthology.orchestra.run.smithi084.stdout: python3-ceph-argparse 2024-03-29T11:16:15.662 INFO:teuthology.orchestra.run.smithi084.stdout: x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 45 k 2024-03-29T11:16:15.662 INFO:teuthology.orchestra.run.smithi084.stdout: python3-ceph-common x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 131 k 2024-03-29T11:16:15.662 INFO:teuthology.orchestra.run.smithi084.stdout: python3-cephfs x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 162 k 2024-03-29T11:16:15.662 INFO:teuthology.orchestra.run.smithi084.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-29T11:16:15.662 INFO:teuthology.orchestra.run.smithi084.stdout: python3-rados x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 320 k 2024-03-29T11:16:15.663 INFO:teuthology.orchestra.run.smithi084.stdout: python3-rbd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 298 k 2024-03-29T11:16:15.663 INFO:teuthology.orchestra.run.smithi084.stdout: python3-rgw x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 99 k 2024-03-29T11:16:15.663 INFO:teuthology.orchestra.run.smithi084.stdout: qatlib x86_64 24.02.0-1.el9 CentOS-AppStream 220 k 2024-03-29T11:16:15.663 INFO:teuthology.orchestra.run.smithi084.stdout: qatzip-libs x86_64 1.2.0-1.el9 CentOS-AppStream 46 k 2024-03-29T11:16:15.663 INFO:teuthology.orchestra.run.smithi084.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-29T11:16:15.663 INFO:teuthology.orchestra.run.smithi084.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-29T11:16:15.663 INFO:teuthology.orchestra.run.smithi084.stdout:Installing weak dependencies: 2024-03-29T11:16:15.663 INFO:teuthology.orchestra.run.smithi084.stdout: qatlib-service x86_64 24.02.0-1.el9 CentOS-AppStream 35 k 2024-03-29T11:16:15.663 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-03-29T11:16:15.663 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction Summary 2024-03-29T11:16:15.663 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2024-03-29T11:16:15.663 INFO:teuthology.orchestra.run.smithi084.stdout:Install 30 Packages 2024-03-29T11:16:15.663 INFO:teuthology.orchestra.run.smithi084.stdout:Upgrade 2 Packages 2024-03-29T11:16:15.663 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-03-29T11:16:15.664 INFO:teuthology.orchestra.run.smithi084.stdout:Total download size: 58 M 2024-03-29T11:16:15.664 INFO:teuthology.orchestra.run.smithi084.stdout:Downloading Packages: 2024-03-29T11:16:17.141 INFO:teuthology.orchestra.run.smithi084.stdout:(1/32): ceph-base-19.0.0-1439.g5dcb190d.el9.x86 4.1 MB/s | 5.3 MB 00:01 2024-03-29T11:16:17.200 INFO:teuthology.orchestra.run.smithi084.stdout:(2/32): ceph-selinux-19.0.0-1439.g5dcb190d.el9. 432 kB/s | 25 kB 00:00 2024-03-29T11:16:17.366 INFO:teuthology.orchestra.run.smithi084.stdout:(3/32): libcephfs2-19.0.0-1439.g5dcb190d.el9.x8 4.4 MB/s | 742 kB 00:00 2024-03-29T11:16:17.517 INFO:teuthology.orchestra.run.smithi084.stdout:(4/32): libradosstriper1-19.0.0-1439.g5dcb190d. 3.2 MB/s | 498 kB 00:00 2024-03-29T11:16:17.629 INFO:teuthology.orchestra.run.smithi179.stdout:lab-extras 39 kB/s | 1.7 kB 00:00 2024-03-29T11:16:18.150 INFO:teuthology.orchestra.run.smithi084.stdout:(5/32): librgw2-19.0.0-1439.g5dcb190d.el9.x86_6 8.1 MB/s | 5.1 MB 00:00 2024-03-29T11:16:18.200 INFO:teuthology.orchestra.run.smithi084.stdout:(6/32): python3-ceph-argparse-19.0.0-1439.g5dcb 906 kB/s | 45 kB 00:00 2024-03-29T11:16:18.267 INFO:teuthology.orchestra.run.smithi084.stdout:(7/32): python3-ceph-common-19.0.0-1439.g5dcb19 1.9 MB/s | 131 kB 00:00 2024-03-29T11:16:18.368 INFO:teuthology.orchestra.run.smithi084.stdout:(8/32): python3-cephfs-19.0.0-1439.g5dcb190d.el 1.6 MB/s | 162 kB 00:00 2024-03-29T11:16:18.735 INFO:teuthology.orchestra.run.smithi084.stdout:(9/32): python3-rados-19.0.0-1439.g5dcb190d.el9 874 kB/s | 320 kB 00:00 2024-03-29T11:16:18.835 INFO:teuthology.orchestra.run.smithi084.stdout:(10/32): python3-rbd-19.0.0-1439.g5dcb190d.el9. 2.9 MB/s | 298 kB 00:00 2024-03-29T11:16:18.902 INFO:teuthology.orchestra.run.smithi084.stdout:(11/32): python3-rgw-19.0.0-1439.g5dcb190d.el9. 1.5 MB/s | 99 kB 00:00 2024-03-29T11:16:19.010 INFO:teuthology.orchestra.run.smithi084.stdout:(12/32): boost-program-options-1.75.0-8.el9.x86 977 kB/s | 106 kB 00:00 2024-03-29T11:16:19.069 INFO:teuthology.orchestra.run.smithi084.stdout:(13/32): libpmemobj-1.12.1-1.el9.x86_64.rpm 2.7 MB/s | 159 kB 00:00 2024-03-29T11:16:19.102 INFO:teuthology.orchestra.run.smithi084.stdout:(14/32): librabbitmq-0.11.0-7.el9.x86_64.rpm 1.3 MB/s | 44 kB 00:00 2024-03-29T11:16:19.169 INFO:teuthology.orchestra.run.smithi084.stdout:(15/32): librdkafka-1.6.1-102.el9.x86_64.rpm 9.7 MB/s | 661 kB 00:00 2024-03-29T11:16:19.228 INFO:teuthology.orchestra.run.smithi084.stdout:(16/32): lttng-ust-2.12.0-6.el9.x86_64.rpm 4.7 MB/s | 282 kB 00:00 2024-03-29T11:16:19.287 INFO:teuthology.orchestra.run.smithi084.stdout:(17/32): python3-prettytable-0.7.2-27.el9.noarc 708 kB/s | 41 kB 00:00 2024-03-29T11:16:19.345 INFO:teuthology.orchestra.run.smithi084.stdout:(18/32): qatlib-24.02.0-1.el9.x86_64.rpm 3.7 MB/s | 220 kB 00:00 2024-03-29T11:16:19.387 INFO:teuthology.orchestra.run.smithi084.stdout:(19/32): qatlib-service-24.02.0-1.el9.x86_64.rp 834 kB/s | 35 kB 00:00 2024-03-29T11:16:19.429 INFO:teuthology.orchestra.run.smithi084.stdout:(20/32): qatzip-libs-1.2.0-1.el9.x86_64.rpm 1.1 MB/s | 46 kB 00:00 2024-03-29T11:16:19.763 INFO:teuthology.orchestra.run.smithi084.stdout:(21/32): gperftools-libs-2.9.1-2.el9.x86_64.rpm 927 kB/s | 309 kB 00:00 2024-03-29T11:16:19.963 INFO:teuthology.orchestra.run.smithi084.stdout:(22/32): ceph-radosgw-19.0.0-1439.g5dcb190d.el9 2.4 MB/s | 9.7 MB 00:04 2024-03-29T11:16:20.080 INFO:teuthology.orchestra.run.smithi084.stdout:(23/32): libarrow-9.0.0-10.el9.x86_64.rpm 14 MB/s | 4.4 MB 00:00 2024-03-29T11:16:20.105 INFO:teuthology.orchestra.run.smithi084.stdout:(24/32): libarrow-doc-9.0.0-10.el9.noarch.rpm 186 kB/s | 26 kB 00:00 2024-03-29T11:16:20.139 INFO:teuthology.orchestra.run.smithi084.stdout:(25/32): liboath-2.6.7-2.el9.x86_64.rpm 833 kB/s | 49 kB 00:00 2024-03-29T11:16:20.164 INFO:teuthology.orchestra.run.smithi084.stdout:(26/32): libunwind-1.6.2-1.el9.x86_64.rpm 1.1 MB/s | 67 kB 00:00 2024-03-29T11:16:20.222 INFO:teuthology.orchestra.run.smithi084.stdout:(27/32): parquet-libs-9.0.0-10.el9.x86_64.rpm 9.8 MB/s | 838 kB 00:00 2024-03-29T11:16:20.231 INFO:teuthology.orchestra.run.smithi179.stdout:Dependencies resolved. 2024-03-29T11:16:20.237 INFO:teuthology.orchestra.run.smithi179.stdout:================================================================================ 2024-03-29T11:16:20.237 INFO:teuthology.orchestra.run.smithi179.stdout: Package Arch Version Repository Size 2024-03-29T11:16:20.237 INFO:teuthology.orchestra.run.smithi179.stdout:================================================================================ 2024-03-29T11:16:20.237 INFO:teuthology.orchestra.run.smithi179.stdout:Installing: 2024-03-29T11:16:20.238 INFO:teuthology.orchestra.run.smithi179.stdout: ceph-radosgw x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 9.7 M 2024-03-29T11:16:20.238 INFO:teuthology.orchestra.run.smithi179.stdout:Upgrading: 2024-03-29T11:16:20.238 INFO:teuthology.orchestra.run.smithi179.stdout: librados2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.4 M 2024-03-29T11:16:20.238 INFO:teuthology.orchestra.run.smithi179.stdout: librbd1 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.1 M 2024-03-29T11:16:20.238 INFO:teuthology.orchestra.run.smithi179.stdout:Installing dependencies: 2024-03-29T11:16:20.238 INFO:teuthology.orchestra.run.smithi179.stdout: boost-program-options 2024-03-29T11:16:20.238 INFO:teuthology.orchestra.run.smithi179.stdout: x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-29T11:16:20.238 INFO:teuthology.orchestra.run.smithi179.stdout: ceph-base x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.3 M 2024-03-29T11:16:20.238 INFO:teuthology.orchestra.run.smithi179.stdout: ceph-common x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 20 M 2024-03-29T11:16:20.238 INFO:teuthology.orchestra.run.smithi179.stdout: ceph-selinux x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 25 k 2024-03-29T11:16:20.238 INFO:teuthology.orchestra.run.smithi179.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-29T11:16:20.239 INFO:teuthology.orchestra.run.smithi179.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-29T11:16:20.239 INFO:teuthology.orchestra.run.smithi179.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-29T11:16:20.239 INFO:teuthology.orchestra.run.smithi179.stdout: libcephfs2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 742 k 2024-03-29T11:16:20.239 INFO:teuthology.orchestra.run.smithi179.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-29T11:16:20.239 INFO:teuthology.orchestra.run.smithi179.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-29T11:16:20.239 INFO:teuthology.orchestra.run.smithi179.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-29T11:16:20.239 INFO:teuthology.orchestra.run.smithi179.stdout: libradosstriper1 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 498 k 2024-03-29T11:16:20.239 INFO:teuthology.orchestra.run.smithi179.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-29T11:16:20.239 INFO:teuthology.orchestra.run.smithi179.stdout: librgw2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.1 M 2024-03-29T11:16:20.239 INFO:teuthology.orchestra.run.smithi179.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-29T11:16:20.239 INFO:teuthology.orchestra.run.smithi179.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-29T11:16:20.239 INFO:teuthology.orchestra.run.smithi179.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-29T11:16:20.240 INFO:teuthology.orchestra.run.smithi179.stdout: python3-ceph-argparse 2024-03-29T11:16:20.240 INFO:teuthology.orchestra.run.smithi179.stdout: x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 45 k 2024-03-29T11:16:20.240 INFO:teuthology.orchestra.run.smithi179.stdout: python3-ceph-common x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 131 k 2024-03-29T11:16:20.240 INFO:teuthology.orchestra.run.smithi179.stdout: python3-cephfs x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 162 k 2024-03-29T11:16:20.240 INFO:teuthology.orchestra.run.smithi179.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-29T11:16:20.240 INFO:teuthology.orchestra.run.smithi179.stdout: python3-rados x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 320 k 2024-03-29T11:16:20.240 INFO:teuthology.orchestra.run.smithi179.stdout: python3-rbd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 298 k 2024-03-29T11:16:20.240 INFO:teuthology.orchestra.run.smithi179.stdout: python3-rgw x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 99 k 2024-03-29T11:16:20.240 INFO:teuthology.orchestra.run.smithi179.stdout: qatlib x86_64 24.02.0-1.el9 CentOS-AppStream 220 k 2024-03-29T11:16:20.240 INFO:teuthology.orchestra.run.smithi179.stdout: qatzip-libs x86_64 1.2.0-1.el9 CentOS-AppStream 46 k 2024-03-29T11:16:20.240 INFO:teuthology.orchestra.run.smithi179.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-29T11:16:20.240 INFO:teuthology.orchestra.run.smithi179.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-29T11:16:20.240 INFO:teuthology.orchestra.run.smithi179.stdout:Installing weak dependencies: 2024-03-29T11:16:20.240 INFO:teuthology.orchestra.run.smithi179.stdout: qatlib-service x86_64 24.02.0-1.el9 CentOS-AppStream 35 k 2024-03-29T11:16:20.240 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-29T11:16:20.241 INFO:teuthology.orchestra.run.smithi179.stdout:Transaction Summary 2024-03-29T11:16:20.241 INFO:teuthology.orchestra.run.smithi179.stdout:================================================================================ 2024-03-29T11:16:20.241 INFO:teuthology.orchestra.run.smithi179.stdout:Install 30 Packages 2024-03-29T11:16:20.241 INFO:teuthology.orchestra.run.smithi179.stdout:Upgrade 2 Packages 2024-03-29T11:16:20.241 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-29T11:16:20.241 INFO:teuthology.orchestra.run.smithi179.stdout:Total download size: 58 M 2024-03-29T11:16:20.241 INFO:teuthology.orchestra.run.smithi179.stdout:Downloading Packages: 2024-03-29T11:16:20.248 INFO:teuthology.orchestra.run.smithi084.stdout:(28/32): re2-20211101-3.el9.x86_64.rpm 2.2 MB/s | 192 kB 00:00 2024-03-29T11:16:20.323 INFO:teuthology.orchestra.run.smithi084.stdout:(29/32): thrift-0.15.0-2.el9.x86_64.rpm 16 MB/s | 1.6 MB 00:00 2024-03-29T11:16:21.099 INFO:teuthology.orchestra.run.smithi084.stdout:(30/32): librbd1-19.0.0-1439.g5dcb190d.el9.x86_ 4.0 MB/s | 3.1 MB 00:00 2024-03-29T11:16:21.799 INFO:teuthology.orchestra.run.smithi084.stdout:(31/32): librados2-19.0.0-1439.g5dcb190d.el9.x8 2.2 MB/s | 3.4 MB 00:01 2024-03-29T11:16:23.046 INFO:teuthology.orchestra.run.smithi179.stdout:(1/32): ceph-base-19.0.0-1439.g5dcb190d.el9.x86 1.9 MB/s | 5.3 MB 00:02 2024-03-29T11:16:23.130 INFO:teuthology.orchestra.run.smithi179.stdout:(2/32): ceph-selinux-19.0.0-1439.g5dcb190d.el9. 302 kB/s | 25 kB 00:00 2024-03-29T11:16:23.422 INFO:teuthology.orchestra.run.smithi179.stdout:(3/32): libcephfs2-19.0.0-1439.g5dcb190d.el9.x8 2.5 MB/s | 742 kB 00:00 2024-03-29T11:16:23.614 INFO:teuthology.orchestra.run.smithi179.stdout:(4/32): libradosstriper1-19.0.0-1439.g5dcb190d. 2.5 MB/s | 498 kB 00:00 2024-03-29T11:16:25.164 INFO:teuthology.orchestra.run.smithi179.stdout:(5/32): librgw2-19.0.0-1439.g5dcb190d.el9.x86_6 3.3 MB/s | 5.1 MB 00:01 2024-03-29T11:16:25.264 INFO:teuthology.orchestra.run.smithi179.stdout:(6/32): python3-ceph-argparse-19.0.0-1439.g5dcb 452 kB/s | 45 kB 00:00 2024-03-29T11:16:25.356 INFO:teuthology.orchestra.run.smithi179.stdout:(7/32): python3-ceph-common-19.0.0-1439.g5dcb19 1.4 MB/s | 131 kB 00:00 2024-03-29T11:16:25.448 INFO:teuthology.orchestra.run.smithi179.stdout:(8/32): python3-cephfs-19.0.0-1439.g5dcb190d.el 1.7 MB/s | 162 kB 00:00 2024-03-29T11:16:25.690 INFO:teuthology.orchestra.run.smithi179.stdout:(9/32): python3-rados-19.0.0-1439.g5dcb190d.el9 1.3 MB/s | 320 kB 00:00 2024-03-29T11:16:25.882 INFO:teuthology.orchestra.run.smithi179.stdout:(10/32): python3-rbd-19.0.0-1439.g5dcb190d.el9. 1.5 MB/s | 298 kB 00:00 2024-03-29T11:16:25.983 INFO:teuthology.orchestra.run.smithi179.stdout:(11/32): python3-rgw-19.0.0-1439.g5dcb190d.el9. 993 kB/s | 99 kB 00:00 2024-03-29T11:16:26.091 INFO:teuthology.orchestra.run.smithi179.stdout:(12/32): boost-program-options-1.75.0-8.el9.x86 979 kB/s | 106 kB 00:00 2024-03-29T11:16:26.133 INFO:teuthology.orchestra.run.smithi179.stdout:(13/32): libpmemobj-1.12.1-1.el9.x86_64.rpm 3.7 MB/s | 159 kB 00:00 2024-03-29T11:16:26.167 INFO:teuthology.orchestra.run.smithi179.stdout:(14/32): librabbitmq-0.11.0-7.el9.x86_64.rpm 1.3 MB/s | 44 kB 00:00 2024-03-29T11:16:26.217 INFO:teuthology.orchestra.run.smithi179.stdout:(15/32): librdkafka-1.6.1-102.el9.x86_64.rpm 13 MB/s | 661 kB 00:00 2024-03-29T11:16:26.259 INFO:teuthology.orchestra.run.smithi179.stdout:(16/32): lttng-ust-2.12.0-6.el9.x86_64.rpm 6.6 MB/s | 282 kB 00:00 2024-03-29T11:16:26.292 INFO:teuthology.orchestra.run.smithi179.stdout:(17/32): python3-prettytable-0.7.2-27.el9.noarc 1.2 MB/s | 41 kB 00:00 2024-03-29T11:16:26.334 INFO:teuthology.orchestra.run.smithi179.stdout:(18/32): qatlib-24.02.0-1.el9.x86_64.rpm 5.1 MB/s | 220 kB 00:00 2024-03-29T11:16:26.368 INFO:teuthology.orchestra.run.smithi179.stdout:(19/32): qatlib-service-24.02.0-1.el9.x86_64.rp 1.0 MB/s | 35 kB 00:00 2024-03-29T11:16:26.402 INFO:teuthology.orchestra.run.smithi179.stdout:(20/32): qatzip-libs-1.2.0-1.el9.x86_64.rpm 1.3 MB/s | 46 kB 00:00 2024-03-29T11:16:26.660 INFO:teuthology.orchestra.run.smithi179.stdout:(21/32): ceph-radosgw-19.0.0-1439.g5dcb190d.el9 1.5 MB/s | 9.7 MB 00:06 2024-03-29T11:16:26.902 INFO:teuthology.orchestra.run.smithi179.stdout:(22/32): gperftools-libs-2.9.1-2.el9.x86_64.rpm 617 kB/s | 309 kB 00:00 2024-03-29T11:16:26.977 INFO:teuthology.orchestra.run.smithi179.stdout:(23/32): libarrow-doc-9.0.0-10.el9.noarch.rpm 347 kB/s | 26 kB 00:00 2024-03-29T11:16:27.053 INFO:teuthology.orchestra.run.smithi179.stdout:(24/32): liboath-2.6.7-2.el9.x86_64.rpm 648 kB/s | 49 kB 00:00 2024-03-29T11:16:27.136 INFO:teuthology.orchestra.run.smithi179.stdout:(25/32): libunwind-1.6.2-1.el9.x86_64.rpm 810 kB/s | 67 kB 00:00 2024-03-29T11:16:27.312 INFO:teuthology.orchestra.run.smithi179.stdout:(26/32): parquet-libs-9.0.0-10.el9.x86_64.rpm 4.7 MB/s | 838 kB 00:00 2024-03-29T11:16:27.387 INFO:teuthology.orchestra.run.smithi179.stdout:(27/32): libarrow-9.0.0-10.el9.x86_64.rpm 6.1 MB/s | 4.4 MB 00:00 2024-03-29T11:16:27.412 INFO:teuthology.orchestra.run.smithi179.stdout:(28/32): re2-20211101-3.el9.x86_64.rpm 1.9 MB/s | 192 kB 00:00 2024-03-29T11:16:27.496 INFO:teuthology.orchestra.run.smithi179.stdout:(29/32): thrift-0.15.0-2.el9.x86_64.rpm 15 MB/s | 1.6 MB 00:00 2024-03-29T11:16:28.954 INFO:teuthology.orchestra.run.smithi179.stdout:(30/32): librados2-19.0.0-1439.g5dcb190d.el9.x8 2.2 MB/s | 3.4 MB 00:01 2024-03-29T11:16:30.621 INFO:teuthology.orchestra.run.smithi179.stdout:(31/32): ceph-common-19.0.0-1439.g5dcb190d.el9. 1.9 MB/s | 20 MB 00:10 2024-03-29T11:16:30.647 INFO:teuthology.orchestra.run.smithi084.stdout:(32/32): ceph-common-19.0.0-1439.g5dcb190d.el9. 1.3 MB/s | 20 MB 00:14 2024-03-29T11:16:30.652 INFO:teuthology.orchestra.run.smithi084.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:16:30.652 INFO:teuthology.orchestra.run.smithi084.stdout:Total 3.9 MB/s | 58 MB 00:14 2024-03-29T11:16:30.652 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction check 2024-03-29T11:16:30.690 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction check succeeded. 2024-03-29T11:16:30.690 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction test 2024-03-29T11:16:31.183 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction test succeeded. 2024-03-29T11:16:31.184 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction 2024-03-29T11:16:32.604 INFO:teuthology.orchestra.run.smithi179.stdout:(32/32): librbd1-19.0.0-1439.g5dcb190d.el9.x86_ 627 kB/s | 3.1 MB 00:05 2024-03-29T11:16:32.610 INFO:teuthology.orchestra.run.smithi179.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:16:32.610 INFO:teuthology.orchestra.run.smithi179.stdout:Total 4.7 MB/s | 58 MB 00:12 2024-03-29T11:16:32.611 INFO:teuthology.orchestra.run.smithi179.stdout:Running transaction check 2024-03-29T11:16:32.648 INFO:teuthology.orchestra.run.smithi179.stdout:Transaction check succeeded. 2024-03-29T11:16:32.649 INFO:teuthology.orchestra.run.smithi179.stdout:Running transaction test 2024-03-29T11:16:32.649 INFO:teuthology.orchestra.run.smithi084.stdout: Preparing : 1/1 2024-03-29T11:16:32.740 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/34 2024-03-29T11:16:32.768 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/34 2024-03-29T11:16:32.794 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/34 2024-03-29T11:16:32.817 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-03-29T11:16:32.833 INFO:teuthology.orchestra.run.smithi084.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-03-29T11:16:33.026 INFO:teuthology.orchestra.run.smithi084.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 5/34 2024-03-29T11:16:33.080 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 5/34 2024-03-29T11:16:33.119 INFO:teuthology.orchestra.run.smithi179.stdout:Transaction test succeeded. 2024-03-29T11:16:33.119 INFO:teuthology.orchestra.run.smithi179.stdout:Running transaction 2024-03-29T11:16:33.401 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/34 2024-03-29T11:16:33.462 INFO:teuthology.orchestra.run.smithi084.stdout: Upgrading : librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-29T11:16:33.553 INFO:teuthology.orchestra.run.smithi084.stdout: Running scriptlet: librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-29T11:16:33.640 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 8/34 2024-03-29T11:16:33.672 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 9/34 2024-03-29T11:16:33.717 INFO:teuthology.orchestra.run.smithi084.stdout: Running scriptlet: libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 9/34 2024-03-29T11:16:33.746 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : liboath-2.6.7-2.el9.x86_64 10/34 2024-03-29T11:16:33.790 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9. 11/34 2024-03-29T11:16:33.859 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 12/34 2024-03-29T11:16:33.882 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 13/34 2024-03-29T11:16:33.929 INFO:teuthology.orchestra.run.smithi084.stdout: Running scriptlet: libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 13/34 2024-03-29T11:16:33.964 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 14/34 2024-03-29T11:16:34.072 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : re2-1:20211101-3.el9.x86_64 15/34 2024-03-29T11:16:34.106 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 16/34 2024-03-29T11:16:34.163 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/34 2024-03-29T11:16:34.196 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 18/34 2024-03-29T11:16:34.225 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 19/34 2024-03-29T11:16:34.258 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 20/34 2024-03-29T11:16:34.573 INFO:teuthology.orchestra.run.smithi179.stdout: Preparing : 1/1 2024-03-29T11:16:34.574 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 21/34 2024-03-29T11:16:34.605 INFO:teuthology.orchestra.run.smithi084.stdout: Upgrading : librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 22/34 2024-03-29T11:16:34.654 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/34 2024-03-29T11:16:34.684 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/34 2024-03-29T11:16:34.695 INFO:teuthology.orchestra.run.smithi084.stdout: Running scriptlet: librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 22/34 2024-03-29T11:16:34.709 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/34 2024-03-29T11:16:34.726 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 23/34 2024-03-29T11:16:34.740 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-03-29T11:16:34.757 INFO:teuthology.orchestra.run.smithi179.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-03-29T11:16:34.770 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 24/34 2024-03-29T11:16:34.899 INFO:teuthology.orchestra.run.smithi179.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 5/34 2024-03-29T11:16:34.954 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 5/34 2024-03-29T11:16:35.218 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 25/34 2024-03-29T11:16:35.241 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 26/34 2024-03-29T11:16:35.271 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/34 2024-03-29T11:16:35.295 INFO:teuthology.orchestra.run.smithi084.stdout: Running scriptlet: librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 26/34 2024-03-29T11:16:35.362 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 27/34 2024-03-29T11:16:35.369 INFO:teuthology.orchestra.run.smithi179.stdout: Upgrading : librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-29T11:16:35.393 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x8 28/34 2024-03-29T11:16:35.460 INFO:teuthology.orchestra.run.smithi179.stdout: Running scriptlet: librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-29T11:16:35.547 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 8/34 2024-03-29T11:16:35.571 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 9/34 2024-03-29T11:16:35.642 INFO:teuthology.orchestra.run.smithi179.stdout: Running scriptlet: libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 9/34 2024-03-29T11:16:35.678 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : liboath-2.6.7-2.el9.x86_64 10/34 2024-03-29T11:16:35.722 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9. 11/34 2024-03-29T11:16:35.789 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 12/34 2024-03-29T11:16:35.815 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 13/34 2024-03-29T11:16:35.878 INFO:teuthology.orchestra.run.smithi179.stdout: Running scriptlet: libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 13/34 2024-03-29T11:16:35.913 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 14/34 2024-03-29T11:16:36.019 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : re2-1:20211101-3.el9.x86_64 15/34 2024-03-29T11:16:36.048 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 16/34 2024-03-29T11:16:36.088 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/34 2024-03-29T11:16:36.121 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 18/34 2024-03-29T11:16:36.150 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 19/34 2024-03-29T11:16:36.183 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 20/34 2024-03-29T11:16:36.504 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 21/34 2024-03-29T11:16:36.529 INFO:teuthology.orchestra.run.smithi179.stdout: Upgrading : librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 22/34 2024-03-29T11:16:36.620 INFO:teuthology.orchestra.run.smithi179.stdout: Running scriptlet: librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 22/34 2024-03-29T11:16:36.659 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 23/34 2024-03-29T11:16:36.730 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 24/34 2024-03-29T11:16:37.186 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 25/34 2024-03-29T11:16:37.215 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 26/34 2024-03-29T11:16:37.278 INFO:teuthology.orchestra.run.smithi179.stdout: Running scriptlet: librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 26/34 2024-03-29T11:16:37.345 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 27/34 2024-03-29T11:16:37.376 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x8 28/34 2024-03-29T11:16:37.591 INFO:teuthology.orchestra.run.smithi084.stdout: Running scriptlet: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-29T11:16:37.631 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-29T11:16:38.174 INFO:teuthology.orchestra.run.smithi084.stdout: Running scriptlet: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-29T11:16:38.209 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 30/34 2024-03-29T11:16:38.299 INFO:teuthology.orchestra.run.smithi084.stdout: Running scriptlet: ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 30/34 2024-03-29T11:16:38.299 INFO:teuthology.orchestra.run.smithi084.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-29T11:16:38.299 INFO:teuthology.orchestra.run.smithi084.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-29T11:16:38.299 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-03-29T11:16:38.335 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-29T11:16:39.504 INFO:teuthology.orchestra.run.smithi179.stdout: Running scriptlet: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-29T11:16:39.539 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-29T11:16:40.080 INFO:teuthology.orchestra.run.smithi179.stdout: Running scriptlet: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-29T11:16:40.134 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 30/34 2024-03-29T11:16:40.212 INFO:teuthology.orchestra.run.smithi179.stdout: Running scriptlet: ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 30/34 2024-03-29T11:16:40.213 INFO:teuthology.orchestra.run.smithi179.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-29T11:16:40.213 INFO:teuthology.orchestra.run.smithi179.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-29T11:16:40.213 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-29T11:16:40.252 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-29T11:16:47.476 INFO:teuthology.orchestra.run.smithi084.stdout: Running scriptlet: ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-29T11:16:47.477 INFO:teuthology.orchestra.run.smithi084.stdout:skipping the directory /sys 2024-03-29T11:16:47.477 INFO:teuthology.orchestra.run.smithi084.stdout:skipping the directory /proc 2024-03-29T11:16:47.477 INFO:teuthology.orchestra.run.smithi084.stdout:skipping the directory /mnt 2024-03-29T11:16:47.477 INFO:teuthology.orchestra.run.smithi084.stdout:skipping the directory /var/tmp 2024-03-29T11:16:47.477 INFO:teuthology.orchestra.run.smithi084.stdout:skipping the directory /home 2024-03-29T11:16:47.477 INFO:teuthology.orchestra.run.smithi084.stdout:skipping the directory /root 2024-03-29T11:16:47.477 INFO:teuthology.orchestra.run.smithi084.stdout:skipping the directory /tmp 2024-03-29T11:16:47.477 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-03-29T11:16:48.382 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/34 2024-03-29T11:16:48.406 INFO:teuthology.orchestra.run.smithi084.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/34 2024-03-29T11:16:48.407 INFO:teuthology.orchestra.run.smithi084.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T11:16:48.407 INFO:teuthology.orchestra.run.smithi084.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-29T11:16:48.407 INFO:teuthology.orchestra.run.smithi084.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-29T11:16:48.407 INFO:teuthology.orchestra.run.smithi084.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-29T11:16:48.407 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-03-29T11:16:48.410 INFO:teuthology.orchestra.run.smithi084.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-29T11:16:48.485 INFO:teuthology.orchestra.run.smithi084.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-29T11:16:48.485 INFO:teuthology.orchestra.run.smithi084.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-29T11:16:49.594 INFO:teuthology.orchestra.run.smithi084.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-29T11:16:49.645 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/34 2024-03-29T11:16:49.645 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/34 2024-03-29T11:16:49.646 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 3/34 2024-03-29T11:16:49.646 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 4/34 2024-03-29T11:16:49.646 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/34 2024-03-29T11:16:49.646 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 6/34 2024-03-29T11:16:49.646 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-29T11:16:49.646 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9. 8/34 2024-03-29T11:16:49.646 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x8 9/34 2024-03-29T11:16:49.647 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 10/34 2024-03-29T11:16:49.647 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 11/34 2024-03-29T11:16:49.647 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 12/34 2024-03-29T11:16:49.647 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 13/34 2024-03-29T11:16:49.647 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/34 2024-03-29T11:16:49.647 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/34 2024-03-29T11:16:49.647 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/34 2024-03-29T11:16:49.647 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/34 2024-03-29T11:16:49.647 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/34 2024-03-29T11:16:49.648 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/34 2024-03-29T11:16:49.648 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 20/34 2024-03-29T11:16:49.648 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 21/34 2024-03-29T11:16:49.648 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 22/34 2024-03-29T11:16:49.648 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 23/34 2024-03-29T11:16:49.648 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 24/34 2024-03-29T11:16:49.648 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 25/34 2024-03-29T11:16:49.648 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 26/34 2024-03-29T11:16:49.648 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 27/34 2024-03-29T11:16:49.649 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 28/34 2024-03-29T11:16:49.649 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : re2-1:20211101-3.el9.x86_64 29/34 2024-03-29T11:16:49.649 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 30/34 2024-03-29T11:16:49.649 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-29T11:16:49.649 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 32/34 2024-03-29T11:16:49.649 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/34 2024-03-29T11:16:49.754 INFO:teuthology.orchestra.run.smithi179.stdout: Running scriptlet: ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-29T11:16:49.754 INFO:teuthology.orchestra.run.smithi179.stdout:skipping the directory /sys 2024-03-29T11:16:49.754 INFO:teuthology.orchestra.run.smithi179.stdout:skipping the directory /proc 2024-03-29T11:16:49.754 INFO:teuthology.orchestra.run.smithi179.stdout:skipping the directory /mnt 2024-03-29T11:16:49.755 INFO:teuthology.orchestra.run.smithi179.stdout:skipping the directory /var/tmp 2024-03-29T11:16:49.755 INFO:teuthology.orchestra.run.smithi179.stdout:skipping the directory /home 2024-03-29T11:16:49.755 INFO:teuthology.orchestra.run.smithi179.stdout:skipping the directory /root 2024-03-29T11:16:49.755 INFO:teuthology.orchestra.run.smithi179.stdout:skipping the directory /tmp 2024-03-29T11:16:49.755 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-29T11:16:50.657 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/34 2024-03-29T11:16:50.666 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 34/34 2024-03-29T11:16:50.666 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-03-29T11:16:50.666 INFO:teuthology.orchestra.run.smithi084.stdout:Upgraded: 2024-03-29T11:16:50.666 INFO:teuthology.orchestra.run.smithi084.stdout: librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:16:50.666 INFO:teuthology.orchestra.run.smithi084.stdout: librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:16:50.666 INFO:teuthology.orchestra.run.smithi084.stdout:Installed: 2024-03-29T11:16:50.667 INFO:teuthology.orchestra.run.smithi084.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-29T11:16:50.667 INFO:teuthology.orchestra.run.smithi084.stdout: ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:16:50.667 INFO:teuthology.orchestra.run.smithi084.stdout: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:16:50.667 INFO:teuthology.orchestra.run.smithi084.stdout: ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:16:50.667 INFO:teuthology.orchestra.run.smithi084.stdout: ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:16:50.667 INFO:teuthology.orchestra.run.smithi084.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-29T11:16:50.667 INFO:teuthology.orchestra.run.smithi084.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-29T11:16:50.667 INFO:teuthology.orchestra.run.smithi084.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-29T11:16:50.667 INFO:teuthology.orchestra.run.smithi084.stdout: libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:16:50.667 INFO:teuthology.orchestra.run.smithi084.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-29T11:16:50.667 INFO:teuthology.orchestra.run.smithi084.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-29T11:16:50.667 INFO:teuthology.orchestra.run.smithi084.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-29T11:16:50.668 INFO:teuthology.orchestra.run.smithi084.stdout: libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:16:50.668 INFO:teuthology.orchestra.run.smithi084.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-29T11:16:50.668 INFO:teuthology.orchestra.run.smithi084.stdout: librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:16:50.668 INFO:teuthology.orchestra.run.smithi084.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-29T11:16:50.668 INFO:teuthology.orchestra.run.smithi084.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-29T11:16:50.668 INFO:teuthology.orchestra.run.smithi084.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-29T11:16:50.668 INFO:teuthology.orchestra.run.smithi084.stdout: python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:16:50.668 INFO:teuthology.orchestra.run.smithi084.stdout: python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:16:50.668 INFO:teuthology.orchestra.run.smithi084.stdout: python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:16:50.668 INFO:teuthology.orchestra.run.smithi084.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-29T11:16:50.668 INFO:teuthology.orchestra.run.smithi084.stdout: python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:16:50.668 INFO:teuthology.orchestra.run.smithi084.stdout: python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:16:50.668 INFO:teuthology.orchestra.run.smithi084.stdout: python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:16:50.669 INFO:teuthology.orchestra.run.smithi084.stdout: qatlib-24.02.0-1.el9.x86_64 2024-03-29T11:16:50.669 INFO:teuthology.orchestra.run.smithi084.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-03-29T11:16:50.669 INFO:teuthology.orchestra.run.smithi084.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-03-29T11:16:50.669 INFO:teuthology.orchestra.run.smithi084.stdout: re2-1:20211101-3.el9.x86_64 2024-03-29T11:16:50.669 INFO:teuthology.orchestra.run.smithi084.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-29T11:16:50.669 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-03-29T11:16:50.669 INFO:teuthology.orchestra.run.smithi084.stdout:Complete! 2024-03-29T11:16:50.686 INFO:teuthology.orchestra.run.smithi179.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/34 2024-03-29T11:16:50.687 INFO:teuthology.orchestra.run.smithi179.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T11:16:50.687 INFO:teuthology.orchestra.run.smithi179.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-29T11:16:50.687 INFO:teuthology.orchestra.run.smithi179.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-29T11:16:50.687 INFO:teuthology.orchestra.run.smithi179.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-29T11:16:50.687 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-29T11:16:50.689 INFO:teuthology.orchestra.run.smithi179.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-29T11:16:50.762 INFO:teuthology.orchestra.run.smithi179.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-29T11:16:50.763 INFO:teuthology.orchestra.run.smithi179.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-29T11:16:50.949 DEBUG:teuthology.orchestra.run.smithi084:> sudo yum -y install ceph-test 2024-03-29T11:16:51.501 INFO:teuthology.orchestra.run.smithi084.stdout:Last metadata expiration check: 0:00:39 ago on Fri 29 Mar 2024 11:16:12 AM UTC. 2024-03-29T11:16:51.661 INFO:teuthology.orchestra.run.smithi084.stdout:Dependencies resolved. 2024-03-29T11:16:51.662 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2024-03-29T11:16:51.662 INFO:teuthology.orchestra.run.smithi084.stdout: Package Arch Version Repository Size 2024-03-29T11:16:51.662 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2024-03-29T11:16:51.662 INFO:teuthology.orchestra.run.smithi084.stdout:Installing: 2024-03-29T11:16:51.662 INFO:teuthology.orchestra.run.smithi084.stdout: ceph-test x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 47 M 2024-03-29T11:16:51.662 INFO:teuthology.orchestra.run.smithi084.stdout:Installing dependencies: 2024-03-29T11:16:51.662 INFO:teuthology.orchestra.run.smithi084.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-03-29T11:16:51.662 INFO:teuthology.orchestra.run.smithi084.stdout: libcephsqlite x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 163 k 2024-03-29T11:16:51.662 INFO:teuthology.orchestra.run.smithi084.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-03-29T11:16:51.662 INFO:teuthology.orchestra.run.smithi084.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-29T11:16:51.662 INFO:teuthology.orchestra.run.smithi084.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-29T11:16:51.662 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-03-29T11:16:51.662 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction Summary 2024-03-29T11:16:51.663 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2024-03-29T11:16:51.663 INFO:teuthology.orchestra.run.smithi084.stdout:Install 6 Packages 2024-03-29T11:16:51.663 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-03-29T11:16:51.663 INFO:teuthology.orchestra.run.smithi084.stdout:Total download size: 48 M 2024-03-29T11:16:51.663 INFO:teuthology.orchestra.run.smithi084.stdout:Installed size: 204 M 2024-03-29T11:16:51.664 INFO:teuthology.orchestra.run.smithi084.stdout:Downloading Packages: 2024-03-29T11:16:51.842 INFO:teuthology.orchestra.run.smithi179.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-29T11:16:51.842 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/34 2024-03-29T11:16:51.843 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/34 2024-03-29T11:16:51.843 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 3/34 2024-03-29T11:16:51.843 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 4/34 2024-03-29T11:16:51.843 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/34 2024-03-29T11:16:51.843 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 6/34 2024-03-29T11:16:51.843 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-29T11:16:51.843 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9. 8/34 2024-03-29T11:16:51.843 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x8 9/34 2024-03-29T11:16:51.843 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 10/34 2024-03-29T11:16:51.843 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 11/34 2024-03-29T11:16:51.843 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 12/34 2024-03-29T11:16:51.843 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 13/34 2024-03-29T11:16:51.843 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/34 2024-03-29T11:16:51.843 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/34 2024-03-29T11:16:51.844 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/34 2024-03-29T11:16:51.844 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/34 2024-03-29T11:16:51.844 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/34 2024-03-29T11:16:51.844 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/34 2024-03-29T11:16:51.844 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 20/34 2024-03-29T11:16:51.844 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 21/34 2024-03-29T11:16:51.844 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 22/34 2024-03-29T11:16:51.844 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 23/34 2024-03-29T11:16:51.844 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 24/34 2024-03-29T11:16:51.844 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 25/34 2024-03-29T11:16:51.844 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 26/34 2024-03-29T11:16:51.844 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 27/34 2024-03-29T11:16:51.844 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 28/34 2024-03-29T11:16:51.844 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : re2-1:20211101-3.el9.x86_64 29/34 2024-03-29T11:16:51.845 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 30/34 2024-03-29T11:16:51.845 INFO:teuthology.orchestra.run.smithi084.stdout:(1/6): oniguruma-6.9.6-1.el9.5.x86_64.rpm 1.8 MB/s | 217 kB 00:00 2024-03-29T11:16:51.846 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-29T11:16:51.846 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 32/34 2024-03-29T11:16:51.846 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/34 2024-03-29T11:16:51.933 INFO:teuthology.orchestra.run.smithi084.stdout:(2/6): socat-1.7.4.1-5.el9.x86_64.rpm 6.3 MB/s | 300 kB 00:00 2024-03-29T11:16:51.967 INFO:teuthology.orchestra.run.smithi084.stdout:(3/6): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.8 MB/s | 63 kB 00:00 2024-03-29T11:16:52.092 INFO:teuthology.orchestra.run.smithi084.stdout:(4/6): jq-1.6-16.el9.x86_64.rpm 1.5 MB/s | 187 kB 00:00 2024-03-29T11:16:53.009 INFO:teuthology.orchestra.run.smithi084.stdout:(5/6): libcephsqlite-19.0.0-1439.g5dcb190d.el9. 127 kB/s | 163 kB 00:01 2024-03-29T11:16:53.062 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 34/34 2024-03-29T11:16:53.062 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-29T11:16:53.062 INFO:teuthology.orchestra.run.smithi179.stdout:Upgraded: 2024-03-29T11:16:53.062 INFO:teuthology.orchestra.run.smithi179.stdout: librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:16:53.062 INFO:teuthology.orchestra.run.smithi179.stdout: librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:16:53.063 INFO:teuthology.orchestra.run.smithi179.stdout:Installed: 2024-03-29T11:16:53.063 INFO:teuthology.orchestra.run.smithi179.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-29T11:16:53.063 INFO:teuthology.orchestra.run.smithi179.stdout: ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:16:53.063 INFO:teuthology.orchestra.run.smithi179.stdout: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:16:53.063 INFO:teuthology.orchestra.run.smithi179.stdout: ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:16:53.063 INFO:teuthology.orchestra.run.smithi179.stdout: ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:16:53.063 INFO:teuthology.orchestra.run.smithi179.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-29T11:16:53.063 INFO:teuthology.orchestra.run.smithi179.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-29T11:16:53.063 INFO:teuthology.orchestra.run.smithi179.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-29T11:16:53.063 INFO:teuthology.orchestra.run.smithi179.stdout: libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:16:53.063 INFO:teuthology.orchestra.run.smithi179.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-29T11:16:53.063 INFO:teuthology.orchestra.run.smithi179.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-29T11:16:53.063 INFO:teuthology.orchestra.run.smithi179.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-29T11:16:53.063 INFO:teuthology.orchestra.run.smithi179.stdout: libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:16:53.063 INFO:teuthology.orchestra.run.smithi179.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-29T11:16:53.063 INFO:teuthology.orchestra.run.smithi179.stdout: librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:16:53.064 INFO:teuthology.orchestra.run.smithi179.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-29T11:16:53.064 INFO:teuthology.orchestra.run.smithi179.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-29T11:16:53.064 INFO:teuthology.orchestra.run.smithi179.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-29T11:16:53.064 INFO:teuthology.orchestra.run.smithi179.stdout: python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:16:53.064 INFO:teuthology.orchestra.run.smithi179.stdout: python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:16:53.064 INFO:teuthology.orchestra.run.smithi179.stdout: python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:16:53.064 INFO:teuthology.orchestra.run.smithi179.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-29T11:16:53.064 INFO:teuthology.orchestra.run.smithi179.stdout: python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:16:53.064 INFO:teuthology.orchestra.run.smithi179.stdout: python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:16:53.064 INFO:teuthology.orchestra.run.smithi179.stdout: python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:16:53.064 INFO:teuthology.orchestra.run.smithi179.stdout: qatlib-24.02.0-1.el9.x86_64 2024-03-29T11:16:53.064 INFO:teuthology.orchestra.run.smithi179.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-03-29T11:16:53.064 INFO:teuthology.orchestra.run.smithi179.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-03-29T11:16:53.064 INFO:teuthology.orchestra.run.smithi179.stdout: re2-1:20211101-3.el9.x86_64 2024-03-29T11:16:53.064 INFO:teuthology.orchestra.run.smithi179.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-29T11:16:53.064 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-29T11:16:53.065 INFO:teuthology.orchestra.run.smithi179.stdout:Complete! 2024-03-29T11:16:53.349 DEBUG:teuthology.orchestra.run.smithi179:> sudo yum -y install ceph-test 2024-03-29T11:16:53.906 INFO:teuthology.orchestra.run.smithi179.stdout:Last metadata expiration check: 0:00:36 ago on Fri 29 Mar 2024 11:16:17 AM UTC. 2024-03-29T11:16:54.061 INFO:teuthology.orchestra.run.smithi179.stdout:Dependencies resolved. 2024-03-29T11:16:54.062 INFO:teuthology.orchestra.run.smithi179.stdout:================================================================================ 2024-03-29T11:16:54.062 INFO:teuthology.orchestra.run.smithi179.stdout: Package Arch Version Repository Size 2024-03-29T11:16:54.062 INFO:teuthology.orchestra.run.smithi179.stdout:================================================================================ 2024-03-29T11:16:54.062 INFO:teuthology.orchestra.run.smithi179.stdout:Installing: 2024-03-29T11:16:54.063 INFO:teuthology.orchestra.run.smithi179.stdout: ceph-test x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 47 M 2024-03-29T11:16:54.063 INFO:teuthology.orchestra.run.smithi179.stdout:Installing dependencies: 2024-03-29T11:16:54.063 INFO:teuthology.orchestra.run.smithi179.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-03-29T11:16:54.063 INFO:teuthology.orchestra.run.smithi179.stdout: libcephsqlite x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 163 k 2024-03-29T11:16:54.063 INFO:teuthology.orchestra.run.smithi179.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-03-29T11:16:54.063 INFO:teuthology.orchestra.run.smithi179.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-29T11:16:54.063 INFO:teuthology.orchestra.run.smithi179.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-29T11:16:54.063 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-29T11:16:54.063 INFO:teuthology.orchestra.run.smithi179.stdout:Transaction Summary 2024-03-29T11:16:54.063 INFO:teuthology.orchestra.run.smithi179.stdout:================================================================================ 2024-03-29T11:16:54.063 INFO:teuthology.orchestra.run.smithi179.stdout:Install 6 Packages 2024-03-29T11:16:54.063 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-29T11:16:54.064 INFO:teuthology.orchestra.run.smithi179.stdout:Total download size: 48 M 2024-03-29T11:16:54.064 INFO:teuthology.orchestra.run.smithi179.stdout:Installed size: 204 M 2024-03-29T11:16:54.064 INFO:teuthology.orchestra.run.smithi179.stdout:Downloading Packages: 2024-03-29T11:16:54.334 INFO:teuthology.orchestra.run.smithi179.stdout:(1/6): oniguruma-6.9.6-1.el9.5.x86_64.rpm 1.6 MB/s | 217 kB 00:00 2024-03-29T11:16:54.400 INFO:teuthology.orchestra.run.smithi179.stdout:(2/6): socat-1.7.4.1-5.el9.x86_64.rpm 6.9 MB/s | 300 kB 00:00 2024-03-29T11:16:54.425 INFO:teuthology.orchestra.run.smithi179.stdout:(3/6): xmlstarlet-1.6.1-20.el9.x86_64.rpm 2.5 MB/s | 63 kB 00:00 2024-03-29T11:16:54.584 INFO:teuthology.orchestra.run.smithi179.stdout:(4/6): jq-1.6-16.el9.x86_64.rpm 1.2 MB/s | 187 kB 00:00 2024-03-29T11:16:55.243 INFO:teuthology.orchestra.run.smithi179.stdout:(5/6): libcephsqlite-19.0.0-1439.g5dcb190d.el9. 156 kB/s | 163 kB 00:01 2024-03-29T11:17:17.908 INFO:teuthology.orchestra.run.smithi179.stdout:(6/6): ceph-test-19.0.0-1439.g5dcb190d.el9.x86_ 2.0 MB/s | 47 MB 00:23 2024-03-29T11:17:17.910 INFO:teuthology.orchestra.run.smithi179.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:17:17.911 INFO:teuthology.orchestra.run.smithi179.stdout:Total 2.0 MB/s | 48 MB 00:23 2024-03-29T11:17:17.924 INFO:teuthology.orchestra.run.smithi179.stdout:Running transaction check 2024-03-29T11:17:17.944 INFO:teuthology.orchestra.run.smithi179.stdout:Transaction check succeeded. 2024-03-29T11:17:17.945 INFO:teuthology.orchestra.run.smithi179.stdout:Running transaction test 2024-03-29T11:17:18.324 INFO:teuthology.orchestra.run.smithi179.stdout:Transaction test succeeded. 2024-03-29T11:17:18.324 INFO:teuthology.orchestra.run.smithi179.stdout:Running transaction 2024-03-29T11:17:18.901 INFO:teuthology.orchestra.run.smithi179.stdout: Preparing : 1/1 2024-03-29T11:17:18.974 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/6 2024-03-29T11:17:19.016 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 2/6 2024-03-29T11:17:19.056 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-29T11:17:19.109 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : jq-1.6-16.el9.x86_64 4/6 2024-03-29T11:17:19.135 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/6 2024-03-29T11:17:19.321 INFO:teuthology.orchestra.run.smithi084.stdout:(6/6): ceph-test-19.0.0-1439.g5dcb190d.el9.x86_ 1.7 MB/s | 47 MB 00:27 2024-03-29T11:17:19.323 INFO:teuthology.orchestra.run.smithi084.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:17:19.324 INFO:teuthology.orchestra.run.smithi084.stdout:Total 1.7 MB/s | 48 MB 00:27 2024-03-29T11:17:19.337 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction check 2024-03-29T11:17:19.358 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction check succeeded. 2024-03-29T11:17:19.358 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction test 2024-03-29T11:17:19.739 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction test succeeded. 2024-03-29T11:17:19.740 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction 2024-03-29T11:17:20.324 INFO:teuthology.orchestra.run.smithi084.stdout: Preparing : 1/1 2024-03-29T11:17:20.398 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/6 2024-03-29T11:17:20.438 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 2/6 2024-03-29T11:17:20.479 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-29T11:17:20.521 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : jq-1.6-16.el9.x86_64 4/6 2024-03-29T11:17:20.549 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/6 2024-03-29T11:17:23.590 INFO:teuthology.orchestra.run.smithi179.stdout: Running scriptlet: libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/6 2024-03-29T11:17:23.668 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 6/6 2024-03-29T11:17:25.089 INFO:teuthology.orchestra.run.smithi084.stdout: Running scriptlet: libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/6 2024-03-29T11:17:25.179 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 6/6 2024-03-29T11:17:26.087 INFO:teuthology.orchestra.run.smithi179.stdout: Running scriptlet: ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 6/6 2024-03-29T11:17:26.087 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/6 2024-03-29T11:17:26.087 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/6 2024-03-29T11:17:26.087 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-29T11:17:26.087 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 4/6 2024-03-29T11:17:26.087 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 5/6 2024-03-29T11:17:26.449 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : jq-1.6-16.el9.x86_64 6/6 2024-03-29T11:17:26.449 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-29T11:17:26.449 INFO:teuthology.orchestra.run.smithi179.stdout:Installed: 2024-03-29T11:17:26.449 INFO:teuthology.orchestra.run.smithi179.stdout: ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:17:26.449 INFO:teuthology.orchestra.run.smithi179.stdout: jq-1.6-16.el9.x86_64 2024-03-29T11:17:26.449 INFO:teuthology.orchestra.run.smithi179.stdout: libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:17:26.449 INFO:teuthology.orchestra.run.smithi179.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-29T11:17:26.450 INFO:teuthology.orchestra.run.smithi179.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-29T11:17:26.450 INFO:teuthology.orchestra.run.smithi179.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-29T11:17:26.450 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-29T11:17:26.450 INFO:teuthology.orchestra.run.smithi179.stdout:Complete! 2024-03-29T11:17:26.762 DEBUG:teuthology.orchestra.run.smithi179:> sudo yum -y install ceph 2024-03-29T11:17:27.322 INFO:teuthology.orchestra.run.smithi179.stdout:Last metadata expiration check: 0:01:10 ago on Fri 29 Mar 2024 11:16:17 AM UTC. 2024-03-29T11:17:27.477 INFO:teuthology.orchestra.run.smithi179.stdout:Dependencies resolved. 2024-03-29T11:17:27.479 INFO:teuthology.orchestra.run.smithi179.stdout:========================================================================================= 2024-03-29T11:17:27.480 INFO:teuthology.orchestra.run.smithi179.stdout: Package Arch Version Repository Size 2024-03-29T11:17:27.480 INFO:teuthology.orchestra.run.smithi179.stdout:========================================================================================= 2024-03-29T11:17:27.480 INFO:teuthology.orchestra.run.smithi179.stdout:Installing: 2024-03-29T11:17:27.480 INFO:teuthology.orchestra.run.smithi179.stdout: ceph x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 6.6 k 2024-03-29T11:17:27.480 INFO:teuthology.orchestra.run.smithi179.stdout:Installing dependencies: 2024-03-29T11:17:27.480 INFO:teuthology.orchestra.run.smithi179.stdout: ceph-mds x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 2.1 M 2024-03-29T11:17:27.480 INFO:teuthology.orchestra.run.smithi179.stdout: ceph-mgr x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 1.5 M 2024-03-29T11:17:27.480 INFO:teuthology.orchestra.run.smithi179.stdout: ceph-mgr-modules-core noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 247 k 2024-03-29T11:17:27.480 INFO:teuthology.orchestra.run.smithi179.stdout: ceph-mon x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 4.7 M 2024-03-29T11:17:27.480 INFO:teuthology.orchestra.run.smithi179.stdout: ceph-osd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 17 M 2024-03-29T11:17:27.480 INFO:teuthology.orchestra.run.smithi179.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-03-29T11:17:27.480 INFO:teuthology.orchestra.run.smithi179.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-03-29T11:17:27.480 INFO:teuthology.orchestra.run.smithi179.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-03-29T11:17:27.481 INFO:teuthology.orchestra.run.smithi179.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-29T11:17:27.481 INFO:teuthology.orchestra.run.smithi179.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-29T11:17:27.481 INFO:teuthology.orchestra.run.smithi179.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-29T11:17:27.481 INFO:teuthology.orchestra.run.smithi179.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-29T11:17:27.481 INFO:teuthology.orchestra.run.smithi179.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-29T11:17:27.481 INFO:teuthology.orchestra.run.smithi179.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-29T11:17:27.481 INFO:teuthology.orchestra.run.smithi179.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-29T11:17:27.481 INFO:teuthology.orchestra.run.smithi179.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-29T11:17:27.481 INFO:teuthology.orchestra.run.smithi179.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-29T11:17:27.481 INFO:teuthology.orchestra.run.smithi179.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-29T11:17:27.481 INFO:teuthology.orchestra.run.smithi179.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-29T11:17:27.481 INFO:teuthology.orchestra.run.smithi179.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-29T11:17:27.482 INFO:teuthology.orchestra.run.smithi179.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-29T11:17:27.482 INFO:teuthology.orchestra.run.smithi179.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-29T11:17:27.482 INFO:teuthology.orchestra.run.smithi179.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-29T11:17:27.482 INFO:teuthology.orchestra.run.smithi179.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-29T11:17:27.482 INFO:teuthology.orchestra.run.smithi179.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-29T11:17:27.482 INFO:teuthology.orchestra.run.smithi179.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-29T11:17:27.482 INFO:teuthology.orchestra.run.smithi179.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-29T11:17:27.482 INFO:teuthology.orchestra.run.smithi179.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-29T11:17:27.482 INFO:teuthology.orchestra.run.smithi179.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-03-29T11:17:27.482 INFO:teuthology.orchestra.run.smithi179.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-29T11:17:27.482 INFO:teuthology.orchestra.run.smithi179.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-29T11:17:27.482 INFO:teuthology.orchestra.run.smithi179.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-29T11:17:27.482 INFO:teuthology.orchestra.run.smithi179.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-29T11:17:27.483 INFO:teuthology.orchestra.run.smithi179.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-29T11:17:27.483 INFO:teuthology.orchestra.run.smithi179.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-29T11:17:27.483 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-29T11:17:27.483 INFO:teuthology.orchestra.run.smithi179.stdout:Transaction Summary 2024-03-29T11:17:27.483 INFO:teuthology.orchestra.run.smithi179.stdout:========================================================================================= 2024-03-29T11:17:27.483 INFO:teuthology.orchestra.run.smithi179.stdout:Install 36 Packages 2024-03-29T11:17:27.483 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-29T11:17:27.483 INFO:teuthology.orchestra.run.smithi179.stdout:Total download size: 30 M 2024-03-29T11:17:27.484 INFO:teuthology.orchestra.run.smithi179.stdout:Installed size: 107 M 2024-03-29T11:17:27.484 INFO:teuthology.orchestra.run.smithi179.stdout:Downloading Packages: 2024-03-29T11:17:27.501 INFO:teuthology.orchestra.run.smithi084.stdout: Running scriptlet: ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 6/6 2024-03-29T11:17:27.502 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/6 2024-03-29T11:17:27.502 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/6 2024-03-29T11:17:27.502 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-29T11:17:27.502 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 4/6 2024-03-29T11:17:27.502 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 5/6 2024-03-29T11:17:27.877 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : jq-1.6-16.el9.x86_64 6/6 2024-03-29T11:17:27.878 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-03-29T11:17:27.878 INFO:teuthology.orchestra.run.smithi084.stdout:Installed: 2024-03-29T11:17:27.878 INFO:teuthology.orchestra.run.smithi084.stdout: ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:17:27.878 INFO:teuthology.orchestra.run.smithi084.stdout: jq-1.6-16.el9.x86_64 2024-03-29T11:17:27.878 INFO:teuthology.orchestra.run.smithi084.stdout: libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:17:27.878 INFO:teuthology.orchestra.run.smithi084.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-29T11:17:27.878 INFO:teuthology.orchestra.run.smithi084.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-29T11:17:27.878 INFO:teuthology.orchestra.run.smithi084.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-29T11:17:27.878 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-03-29T11:17:27.878 INFO:teuthology.orchestra.run.smithi084.stdout:Complete! 2024-03-29T11:17:28.206 DEBUG:teuthology.orchestra.run.smithi084:> sudo yum -y install ceph 2024-03-29T11:17:28.618 INFO:teuthology.orchestra.run.smithi179.stdout:(1/36): ceph-19.0.0-1439.g5dcb190d.el9.x86_64.r 16 kB/s | 6.6 kB 00:00 2024-03-29T11:17:28.777 INFO:teuthology.orchestra.run.smithi084.stdout:Last metadata expiration check: 0:01:16 ago on Fri 29 Mar 2024 11:16:12 AM UTC. 2024-03-29T11:17:28.931 INFO:teuthology.orchestra.run.smithi084.stdout:Dependencies resolved. 2024-03-29T11:17:28.933 INFO:teuthology.orchestra.run.smithi084.stdout:========================================================================================= 2024-03-29T11:17:28.933 INFO:teuthology.orchestra.run.smithi084.stdout: Package Arch Version Repository Size 2024-03-29T11:17:28.933 INFO:teuthology.orchestra.run.smithi084.stdout:========================================================================================= 2024-03-29T11:17:28.933 INFO:teuthology.orchestra.run.smithi084.stdout:Installing: 2024-03-29T11:17:28.933 INFO:teuthology.orchestra.run.smithi084.stdout: ceph x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 6.6 k 2024-03-29T11:17:28.933 INFO:teuthology.orchestra.run.smithi084.stdout:Installing dependencies: 2024-03-29T11:17:28.933 INFO:teuthology.orchestra.run.smithi084.stdout: ceph-mds x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 2.1 M 2024-03-29T11:17:28.934 INFO:teuthology.orchestra.run.smithi084.stdout: ceph-mgr x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 1.5 M 2024-03-29T11:17:28.934 INFO:teuthology.orchestra.run.smithi084.stdout: ceph-mgr-modules-core noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 247 k 2024-03-29T11:17:28.934 INFO:teuthology.orchestra.run.smithi084.stdout: ceph-mon x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 4.7 M 2024-03-29T11:17:28.934 INFO:teuthology.orchestra.run.smithi084.stdout: ceph-osd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 17 M 2024-03-29T11:17:28.934 INFO:teuthology.orchestra.run.smithi084.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-03-29T11:17:28.934 INFO:teuthology.orchestra.run.smithi084.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-03-29T11:17:28.934 INFO:teuthology.orchestra.run.smithi084.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-03-29T11:17:28.934 INFO:teuthology.orchestra.run.smithi084.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-29T11:17:28.934 INFO:teuthology.orchestra.run.smithi084.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-29T11:17:28.934 INFO:teuthology.orchestra.run.smithi084.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-29T11:17:28.934 INFO:teuthology.orchestra.run.smithi084.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-29T11:17:28.934 INFO:teuthology.orchestra.run.smithi084.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-29T11:17:28.934 INFO:teuthology.orchestra.run.smithi084.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-29T11:17:28.934 INFO:teuthology.orchestra.run.smithi084.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-29T11:17:28.935 INFO:teuthology.orchestra.run.smithi084.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-29T11:17:28.935 INFO:teuthology.orchestra.run.smithi084.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-29T11:17:28.935 INFO:teuthology.orchestra.run.smithi084.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-29T11:17:28.935 INFO:teuthology.orchestra.run.smithi084.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-29T11:17:28.935 INFO:teuthology.orchestra.run.smithi084.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-29T11:17:28.935 INFO:teuthology.orchestra.run.smithi084.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-29T11:17:28.935 INFO:teuthology.orchestra.run.smithi084.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-29T11:17:28.935 INFO:teuthology.orchestra.run.smithi084.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-29T11:17:28.935 INFO:teuthology.orchestra.run.smithi084.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-29T11:17:28.935 INFO:teuthology.orchestra.run.smithi084.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-29T11:17:28.935 INFO:teuthology.orchestra.run.smithi084.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-29T11:17:28.935 INFO:teuthology.orchestra.run.smithi084.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-29T11:17:28.935 INFO:teuthology.orchestra.run.smithi084.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-29T11:17:28.935 INFO:teuthology.orchestra.run.smithi084.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-03-29T11:17:28.936 INFO:teuthology.orchestra.run.smithi084.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-29T11:17:28.936 INFO:teuthology.orchestra.run.smithi084.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-29T11:17:28.936 INFO:teuthology.orchestra.run.smithi084.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-29T11:17:28.936 INFO:teuthology.orchestra.run.smithi084.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-29T11:17:28.936 INFO:teuthology.orchestra.run.smithi084.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-29T11:17:28.936 INFO:teuthology.orchestra.run.smithi084.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-29T11:17:28.936 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-03-29T11:17:28.936 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction Summary 2024-03-29T11:17:28.936 INFO:teuthology.orchestra.run.smithi084.stdout:========================================================================================= 2024-03-29T11:17:28.936 INFO:teuthology.orchestra.run.smithi084.stdout:Install 36 Packages 2024-03-29T11:17:28.936 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-03-29T11:17:28.937 INFO:teuthology.orchestra.run.smithi084.stdout:Total download size: 30 M 2024-03-29T11:17:28.937 INFO:teuthology.orchestra.run.smithi084.stdout:Installed size: 107 M 2024-03-29T11:17:28.937 INFO:teuthology.orchestra.run.smithi084.stdout:Downloading Packages: 2024-03-29T11:17:29.794 INFO:teuthology.orchestra.run.smithi179.stdout:(2/36): ceph-mgr-19.0.0-1439.g5dcb190d.el9.x86_ 963 kB/s | 1.5 MB 00:01 2024-03-29T11:17:29.880 INFO:teuthology.orchestra.run.smithi084.stdout:(1/36): ceph-19.0.0-1439.g5dcb190d.el9.x86_64.r 16 kB/s | 6.6 kB 00:00 2024-03-29T11:17:30.236 INFO:teuthology.orchestra.run.smithi179.stdout:(3/36): ceph-mds-19.0.0-1439.g5dcb190d.el9.x86_ 1.0 MB/s | 2.1 MB 00:02 2024-03-29T11:17:30.444 INFO:teuthology.orchestra.run.smithi179.stdout:(4/36): ceph-mgr-modules-core-19.0.0-1439.g5dcb 1.2 MB/s | 247 kB 00:00 2024-03-29T11:17:30.570 INFO:teuthology.orchestra.run.smithi179.stdout:(5/36): lua-5.4.4-4.el9.x86_64.rpm 1.5 MB/s | 187 kB 00:00 2024-03-29T11:17:30.611 INFO:teuthology.orchestra.run.smithi179.stdout:(6/36): python3-mako-1.1.4-6.el9.noarch.rpm 3.4 MB/s | 144 kB 00:00 2024-03-29T11:17:30.645 INFO:teuthology.orchestra.run.smithi179.stdout:(7/36): python3-markupsafe-1.1.1-12.el9.x86_64. 942 kB/s | 32 kB 00:00 2024-03-29T11:17:30.679 INFO:teuthology.orchestra.run.smithi179.stdout:(8/36): python3-pytz-2021.1-5.el9.noarch.rpm 1.4 MB/s | 47 kB 00:00 2024-03-29T11:17:30.746 INFO:teuthology.orchestra.run.smithi179.stdout:(9/36): python3-toml-0.10.2-6.el9.noarch.rpm 580 kB/s | 39 kB 00:00 2024-03-29T11:17:30.788 INFO:teuthology.orchestra.run.smithi179.stdout:(10/36): python3-cffi-1.14.5-5.el9.x86_64.rpm 5.6 MB/s | 241 kB 00:00 2024-03-29T11:17:30.863 INFO:teuthology.orchestra.run.smithi179.stdout:(11/36): python3-cryptography-36.0.1-4.el9.x86_ 15 MB/s | 1.1 MB 00:00 2024-03-29T11:17:30.905 INFO:teuthology.orchestra.run.smithi179.stdout:(12/36): python3-ply-3.11-14.el9.noarch.rpm 2.4 MB/s | 103 kB 00:00 2024-03-29T11:17:30.955 INFO:teuthology.orchestra.run.smithi179.stdout:(13/36): python3-pycparser-2.20-6.el9.noarch.rp 2.4 MB/s | 124 kB 00:00 2024-03-29T11:17:30.997 INFO:teuthology.orchestra.run.smithi179.stdout:(14/36): python3-urllib3-1.26.5-5.el9.noarch.rp 4.4 MB/s | 187 kB 00:00 2024-03-29T11:17:31.072 INFO:teuthology.orchestra.run.smithi179.stdout:(15/36): python3-requests-2.25.1-8.el9.noarch.r 1.6 MB/s | 125 kB 00:00 2024-03-29T11:17:31.239 INFO:teuthology.orchestra.run.smithi179.stdout:(16/36): lua-devel-5.4.4-4.el9.x86_64.rpm 134 kB/s | 22 kB 00:00 2024-03-29T11:17:31.381 INFO:teuthology.orchestra.run.smithi179.stdout:(17/36): luarocks-3.9.2-1.el9.noarch.rpm 1.0 MB/s | 151 kB 00:00 2024-03-29T11:17:31.397 INFO:teuthology.orchestra.run.smithi084.stdout:(2/36): ceph-mgr-19.0.0-1439.g5dcb190d.el9.x86_ 794 kB/s | 1.5 MB 00:01 2024-03-29T11:17:31.432 INFO:teuthology.orchestra.run.smithi179.stdout:(18/36): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 871 kB/s | 43 kB 00:00 2024-03-29T11:17:31.490 INFO:teuthology.orchestra.run.smithi179.stdout:(19/36): python3-cheroot-8.6.0-4.el9.noarch.rpm 2.9 MB/s | 172 kB 00:00 2024-03-29T11:17:31.557 INFO:teuthology.orchestra.run.smithi179.stdout:(20/36): python3-cherrypy-18.6.1-2.el9.noarch.r 5.3 MB/s | 358 kB 00:00 2024-03-29T11:17:31.591 INFO:teuthology.orchestra.run.smithi179.stdout:(21/36): python3-jaraco-8.2.1-3.el9.noarch.rpm 320 kB/s | 11 kB 00:00 2024-03-29T11:17:31.624 INFO:teuthology.orchestra.run.smithi179.stdout:(22/36): python3-jaraco-classes-3.2.1-5.el9.noa 528 kB/s | 18 kB 00:00 2024-03-29T11:17:31.658 INFO:teuthology.orchestra.run.smithi179.stdout:(23/36): python3-jaraco-collections-3.0.0-8.el9 697 kB/s | 23 kB 00:00 2024-03-29T11:17:31.692 INFO:teuthology.orchestra.run.smithi179.stdout:(24/36): python3-jaraco-functools-3.5.0-2.el9.n 583 kB/s | 19 kB 00:00 2024-03-29T11:17:31.750 INFO:teuthology.orchestra.run.smithi179.stdout:(25/36): ceph-mon-19.0.0-1439.g5dcb190d.el9.x86 1.5 MB/s | 4.7 MB 00:03 2024-03-29T11:17:31.767 INFO:teuthology.orchestra.run.smithi179.stdout:(26/36): python3-jaraco-text-3.2.0-6.el9.noarch 262 kB/s | 20 kB 00:00 2024-03-29T11:17:31.809 INFO:teuthology.orchestra.run.smithi179.stdout:(27/36): python3-more-itertools-8.12.0-2.el9.no 1.8 MB/s | 79 kB 00:00 2024-03-29T11:17:31.851 INFO:teuthology.orchestra.run.smithi179.stdout:(28/36): python3-logutils-0.3.5-21.el9.noarch.r 461 kB/s | 46 kB 00:00 2024-03-29T11:17:31.876 INFO:teuthology.orchestra.run.smithi179.stdout:(29/36): python3-pecan-1.4.2-3.el9.noarch.rpm 4.0 MB/s | 272 kB 00:00 2024-03-29T11:17:31.893 INFO:teuthology.orchestra.run.smithi179.stdout:(30/36): python3-portend-3.1.0-2.el9.noarch.rpm 392 kB/s | 16 kB 00:00 2024-03-29T11:17:31.897 INFO:teuthology.orchestra.run.smithi084.stdout:(3/36): ceph-mds-19.0.0-1439.g5dcb190d.el9.x86_ 898 kB/s | 2.1 MB 00:02 2024-03-29T11:17:31.927 INFO:teuthology.orchestra.run.smithi179.stdout:(31/36): python3-pyOpenSSL-21.0.0-1.el9.noarch. 1.7 MB/s | 90 kB 00:00 2024-03-29T11:17:31.944 INFO:teuthology.orchestra.run.smithi179.stdout:(32/36): python3-tempora-5.0.0-2.el9.noarch.rpm 708 kB/s | 36 kB 00:00 2024-03-29T11:17:31.978 INFO:teuthology.orchestra.run.smithi179.stdout:(33/36): python3-webob-1.8.7-6.el9.noarch.rpm 4.5 MB/s | 230 kB 00:00 2024-03-29T11:17:32.011 INFO:teuthology.orchestra.run.smithi179.stdout:(34/36): python3-zc-lockfile-2.0-10.el9.noarch. 603 kB/s | 20 kB 00:00 2024-03-29T11:17:32.053 INFO:teuthology.orchestra.run.smithi179.stdout:(35/36): python3-werkzeug-2.0.3-3.el9.1.noarch. 3.8 MB/s | 427 kB 00:00 2024-03-29T11:17:32.197 INFO:teuthology.orchestra.run.smithi084.stdout:(4/36): ceph-mgr-modules-core-19.0.0-1439.g5dcb 825 kB/s | 247 kB 00:00 2024-03-29T11:17:32.322 INFO:teuthology.orchestra.run.smithi084.stdout:(5/36): lua-5.4.4-4.el9.x86_64.rpm 1.5 MB/s | 187 kB 00:00 2024-03-29T11:17:32.364 INFO:teuthology.orchestra.run.smithi084.stdout:(6/36): python3-mako-1.1.4-6.el9.noarch.rpm 3.4 MB/s | 144 kB 00:00 2024-03-29T11:17:32.389 INFO:teuthology.orchestra.run.smithi084.stdout:(7/36): python3-markupsafe-1.1.1-12.el9.x86_64. 1.2 MB/s | 32 kB 00:00 2024-03-29T11:17:32.414 INFO:teuthology.orchestra.run.smithi084.stdout:(8/36): python3-pytz-2021.1-5.el9.noarch.rpm 1.9 MB/s | 47 kB 00:00 2024-03-29T11:17:32.456 INFO:teuthology.orchestra.run.smithi084.stdout:(9/36): python3-toml-0.10.2-6.el9.noarch.rpm 927 kB/s | 39 kB 00:00 2024-03-29T11:17:32.498 INFO:teuthology.orchestra.run.smithi084.stdout:(10/36): python3-cffi-1.14.5-5.el9.x86_64.rpm 5.7 MB/s | 241 kB 00:00 2024-03-29T11:17:32.565 INFO:teuthology.orchestra.run.smithi084.stdout:(11/36): python3-cryptography-36.0.1-4.el9.x86_ 17 MB/s | 1.1 MB 00:00 2024-03-29T11:17:32.600 INFO:teuthology.orchestra.run.smithi084.stdout:(12/36): python3-ply-3.11-14.el9.noarch.rpm 3.0 MB/s | 103 kB 00:00 2024-03-29T11:17:32.633 INFO:teuthology.orchestra.run.smithi084.stdout:(13/36): python3-pycparser-2.20-6.el9.noarch.rp 3.6 MB/s | 124 kB 00:00 2024-03-29T11:17:32.666 INFO:teuthology.orchestra.run.smithi084.stdout:(14/36): python3-urllib3-1.26.5-5.el9.noarch.rp 5.5 MB/s | 187 kB 00:00 2024-03-29T11:17:33.425 INFO:teuthology.orchestra.run.smithi084.stdout:(15/36): python3-requests-2.25.1-8.el9.noarch.r 165 kB/s | 125 kB 00:00 2024-03-29T11:17:34.000 INFO:teuthology.orchestra.run.smithi084.stdout:(16/36): lua-devel-5.4.4-4.el9.x86_64.rpm 39 kB/s | 22 kB 00:00 2024-03-29T11:17:34.167 INFO:teuthology.orchestra.run.smithi084.stdout:(17/36): ceph-mon-19.0.0-1439.g5dcb190d.el9.x86 1.1 MB/s | 4.7 MB 00:04 2024-03-29T11:17:34.567 INFO:teuthology.orchestra.run.smithi084.stdout:(18/36): luarocks-3.9.2-1.el9.noarch.rpm 267 kB/s | 151 kB 00:00 2024-03-29T11:17:34.592 INFO:teuthology.orchestra.run.smithi084.stdout:(19/36): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 102 kB/s | 43 kB 00:00 2024-03-29T11:17:34.743 INFO:teuthology.orchestra.run.smithi084.stdout:(20/36): python3-cheroot-8.6.0-4.el9.noarch.rpm 981 kB/s | 172 kB 00:00 2024-03-29T11:17:34.843 INFO:teuthology.orchestra.run.smithi084.stdout:(21/36): python3-jaraco-8.2.1-3.el9.noarch.rpm 107 kB/s | 11 kB 00:00 2024-03-29T11:17:34.952 INFO:teuthology.orchestra.run.smithi084.stdout:(22/36): python3-jaraco-classes-3.2.1-5.el9.noa 163 kB/s | 18 kB 00:00 2024-03-29T11:17:34.977 INFO:teuthology.orchestra.run.smithi084.stdout:(23/36): python3-cherrypy-18.6.1-2.el9.noarch.r 933 kB/s | 358 kB 00:00 2024-03-29T11:17:35.052 INFO:teuthology.orchestra.run.smithi084.stdout:(24/36): python3-jaraco-collections-3.0.0-8.el9 231 kB/s | 23 kB 00:00 2024-03-29T11:17:35.069 INFO:teuthology.orchestra.run.smithi084.stdout:(25/36): python3-jaraco-functools-3.5.0-2.el9.n 211 kB/s | 19 kB 00:00 2024-03-29T11:17:35.144 INFO:teuthology.orchestra.run.smithi084.stdout:(26/36): python3-jaraco-text-3.2.0-6.el9.noarch 214 kB/s | 20 kB 00:00 2024-03-29T11:17:35.169 INFO:teuthology.orchestra.run.smithi084.stdout:(27/36): python3-logutils-0.3.5-21.el9.noarch.r 463 kB/s | 46 kB 00:00 2024-03-29T11:17:35.253 INFO:teuthology.orchestra.run.smithi084.stdout:(28/36): python3-more-itertools-8.12.0-2.el9.no 727 kB/s | 79 kB 00:00 2024-03-29T11:17:35.312 INFO:teuthology.orchestra.run.smithi084.stdout:(29/36): python3-pecan-1.4.2-3.el9.noarch.rpm 1.9 MB/s | 272 kB 00:00 2024-03-29T11:17:35.345 INFO:teuthology.orchestra.run.smithi084.stdout:(30/36): python3-portend-3.1.0-2.el9.noarch.rpm 179 kB/s | 16 kB 00:00 2024-03-29T11:17:35.421 INFO:teuthology.orchestra.run.smithi084.stdout:(31/36): python3-pyOpenSSL-21.0.0-1.el9.noarch. 829 kB/s | 90 kB 00:00 2024-03-29T11:17:35.437 INFO:teuthology.orchestra.run.smithi084.stdout:(32/36): python3-tempora-5.0.0-2.el9.noarch.rpm 390 kB/s | 36 kB 00:00 2024-03-29T11:17:35.554 INFO:teuthology.orchestra.run.smithi084.stdout:(33/36): python3-webob-1.8.7-6.el9.noarch.rpm 1.7 MB/s | 230 kB 00:00 2024-03-29T11:17:35.604 INFO:teuthology.orchestra.run.smithi084.stdout:(34/36): python3-werkzeug-2.0.3-3.el9.1.noarch. 2.5 MB/s | 427 kB 00:00 2024-03-29T11:17:35.646 INFO:teuthology.orchestra.run.smithi084.stdout:(35/36): python3-zc-lockfile-2.0-10.el9.noarch. 218 kB/s | 20 kB 00:00 2024-03-29T11:17:37.403 INFO:teuthology.orchestra.run.smithi179.stdout:(36/36): ceph-osd-19.0.0-1439.g5dcb190d.el9.x86 2.2 MB/s | 17 MB 00:07 2024-03-29T11:17:37.408 INFO:teuthology.orchestra.run.smithi179.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:17:37.408 INFO:teuthology.orchestra.run.smithi179.stdout:Total 3.0 MB/s | 30 MB 00:09 2024-03-29T11:17:37.433 INFO:teuthology.orchestra.run.smithi179.stdout:Running transaction check 2024-03-29T11:17:37.476 INFO:teuthology.orchestra.run.smithi179.stdout:Transaction check succeeded. 2024-03-29T11:17:37.476 INFO:teuthology.orchestra.run.smithi179.stdout:Running transaction test 2024-03-29T11:17:37.893 INFO:teuthology.orchestra.run.smithi179.stdout:Transaction test succeeded. 2024-03-29T11:17:37.894 INFO:teuthology.orchestra.run.smithi179.stdout:Running transaction 2024-03-29T11:17:39.220 INFO:teuthology.orchestra.run.smithi179.stdout: Preparing : 1/1 2024-03-29T11:17:39.280 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/36 2024-03-29T11:17:39.317 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/36 2024-03-29T11:17:39.341 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/36 2024-03-29T11:17:39.403 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/36 2024-03-29T11:17:39.440 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : luarocks-3.9.2-1.el9.noarch 5/36 2024-03-29T11:17:39.472 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/36 2024-03-29T11:17:39.564 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/36 2024-03-29T11:17:39.643 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/36 2024-03-29T11:17:39.683 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 9/36 2024-03-29T11:17:39.755 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/36 2024-03-29T11:17:39.808 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 11/36 2024-03-29T11:17:39.847 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 12/36 2024-03-29T11:17:39.896 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-ply-3.11-14.el9.noarch 13/36 2024-03-29T11:17:39.952 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 14/36 2024-03-29T11:17:40.136 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 15/36 2024-03-29T11:17:40.177 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 16/36 2024-03-29T11:17:40.207 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 17/36 2024-03-29T11:17:40.241 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 18/36 2024-03-29T11:17:40.272 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 19/36 2024-03-29T11:17:40.347 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 20/36 2024-03-29T11:17:40.391 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-29T11:17:40.424 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 22/36 2024-03-29T11:17:40.464 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 23/36 2024-03-29T11:17:40.500 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 24/36 2024-03-29T11:17:40.534 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 25/36 2024-03-29T11:17:40.624 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 26/36 2024-03-29T11:17:40.664 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 27/36 2024-03-29T11:17:40.738 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 28/36 2024-03-29T11:17:40.881 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 29/36 2024-03-29T11:17:41.007 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-29T11:17:41.166 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9. 31/36 2024-03-29T11:17:41.195 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/36 2024-03-29T11:17:41.220 INFO:teuthology.orchestra.run.smithi179.stdout: Running scriptlet: ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/36 2024-03-29T11:17:41.221 INFO:teuthology.orchestra.run.smithi179.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T11:17:41.221 INFO:teuthology.orchestra.run.smithi179.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-29T11:17:41.221 INFO:teuthology.orchestra.run.smithi179.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-29T11:17:41.221 INFO:teuthology.orchestra.run.smithi179.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-29T11:17:41.221 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-29T11:17:42.646 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/36 2024-03-29T11:17:42.672 INFO:teuthology.orchestra.run.smithi179.stdout: Running scriptlet: ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/36 2024-03-29T11:17:42.672 INFO:teuthology.orchestra.run.smithi179.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T11:17:42.672 INFO:teuthology.orchestra.run.smithi179.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-29T11:17:42.673 INFO:teuthology.orchestra.run.smithi179.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-29T11:17:42.673 INFO:teuthology.orchestra.run.smithi179.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-29T11:17:42.673 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-29T11:17:43.113 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 34/36 2024-03-29T11:17:43.136 INFO:teuthology.orchestra.run.smithi179.stdout: Running scriptlet: ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 34/36 2024-03-29T11:17:43.136 INFO:teuthology.orchestra.run.smithi179.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T11:17:43.137 INFO:teuthology.orchestra.run.smithi179.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-29T11:17:43.137 INFO:teuthology.orchestra.run.smithi179.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-29T11:17:43.137 INFO:teuthology.orchestra.run.smithi179.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-29T11:17:43.137 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-29T11:17:43.347 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 35/36 2024-03-29T11:17:43.370 INFO:teuthology.orchestra.run.smithi179.stdout: Running scriptlet: ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 35/36 2024-03-29T11:17:43.371 INFO:teuthology.orchestra.run.smithi179.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T11:17:43.371 INFO:teuthology.orchestra.run.smithi179.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-29T11:17:43.371 INFO:teuthology.orchestra.run.smithi179.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-29T11:17:43.371 INFO:teuthology.orchestra.run.smithi179.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-29T11:17:43.371 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-29T11:17:43.428 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 36/36 2024-03-29T11:17:43.612 INFO:teuthology.orchestra.run.smithi084.stdout:(36/36): ceph-osd-19.0.0-1439.g5dcb190d.el9.x86 1.4 MB/s | 17 MB 00:12 2024-03-29T11:17:43.618 INFO:teuthology.orchestra.run.smithi084.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:17:43.618 INFO:teuthology.orchestra.run.smithi084.stdout:Total 2.0 MB/s | 30 MB 00:14 2024-03-29T11:17:43.642 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction check 2024-03-29T11:17:43.684 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction check succeeded. 2024-03-29T11:17:43.685 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction test 2024-03-29T11:17:44.102 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction test succeeded. 2024-03-29T11:17:44.103 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction 2024-03-29T11:17:45.258 INFO:teuthology.orchestra.run.smithi179.stdout: Running scriptlet: ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 36/36 2024-03-29T11:17:45.258 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/36 2024-03-29T11:17:45.259 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/36 2024-03-29T11:17:45.259 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 3/36 2024-03-29T11:17:45.259 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 4/36 2024-03-29T11:17:45.259 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/36 2024-03-29T11:17:45.259 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9. 6/36 2024-03-29T11:17:45.259 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : lua-5.4.4-4.el9.x86_64 7/36 2024-03-29T11:17:45.259 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 8/36 2024-03-29T11:17:45.259 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 9/36 2024-03-29T11:17:45.259 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 10/36 2024-03-29T11:17:45.259 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 11/36 2024-03-29T11:17:45.259 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 12/36 2024-03-29T11:17:45.259 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 13/36 2024-03-29T11:17:45.259 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-ply-3.11-14.el9.noarch 14/36 2024-03-29T11:17:45.259 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 15/36 2024-03-29T11:17:45.260 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 16/36 2024-03-29T11:17:45.260 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 17/36 2024-03-29T11:17:45.260 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 18/36 2024-03-29T11:17:45.260 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 19/36 2024-03-29T11:17:45.260 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 20/36 2024-03-29T11:17:45.260 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-29T11:17:45.260 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 22/36 2024-03-29T11:17:45.260 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 23/36 2024-03-29T11:17:45.260 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 24/36 2024-03-29T11:17:45.260 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 25/36 2024-03-29T11:17:45.260 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 26/36 2024-03-29T11:17:45.261 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 27/36 2024-03-29T11:17:45.262 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 28/36 2024-03-29T11:17:45.262 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 29/36 2024-03-29T11:17:45.262 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-29T11:17:45.262 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 31/36 2024-03-29T11:17:45.262 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 32/36 2024-03-29T11:17:45.262 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 33/36 2024-03-29T11:17:45.262 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 34/36 2024-03-29T11:17:45.262 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 35/36 2024-03-29T11:17:45.450 INFO:teuthology.orchestra.run.smithi084.stdout: Preparing : 1/1 2024-03-29T11:17:45.522 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/36 2024-03-29T11:17:45.556 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/36 2024-03-29T11:17:45.583 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/36 2024-03-29T11:17:45.654 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/36 2024-03-29T11:17:45.707 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : luarocks-3.9.2-1.el9.noarch 5/36 2024-03-29T11:17:45.740 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/36 2024-03-29T11:17:45.830 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/36 2024-03-29T11:17:45.901 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/36 2024-03-29T11:17:45.942 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 9/36 2024-03-29T11:17:46.013 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/36 2024-03-29T11:17:46.068 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 11/36 2024-03-29T11:17:46.114 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 12/36 2024-03-29T11:17:46.144 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 36/36 2024-03-29T11:17:46.145 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-29T11:17:46.145 INFO:teuthology.orchestra.run.smithi179.stdout:Installed: 2024-03-29T11:17:46.145 INFO:teuthology.orchestra.run.smithi179.stdout: ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:17:46.145 INFO:teuthology.orchestra.run.smithi179.stdout: ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:17:46.145 INFO:teuthology.orchestra.run.smithi179.stdout: ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:17:46.145 INFO:teuthology.orchestra.run.smithi179.stdout: ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:17:46.145 INFO:teuthology.orchestra.run.smithi179.stdout: ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:17:46.145 INFO:teuthology.orchestra.run.smithi179.stdout: ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:17:46.145 INFO:teuthology.orchestra.run.smithi179.stdout: lua-5.4.4-4.el9.x86_64 2024-03-29T11:17:46.145 INFO:teuthology.orchestra.run.smithi179.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-03-29T11:17:46.145 INFO:teuthology.orchestra.run.smithi179.stdout: luarocks-3.9.2-1.el9.noarch 2024-03-29T11:17:46.145 INFO:teuthology.orchestra.run.smithi179.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-29T11:17:46.145 INFO:teuthology.orchestra.run.smithi179.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-29T11:17:46.145 INFO:teuthology.orchestra.run.smithi179.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-29T11:17:46.146 INFO:teuthology.orchestra.run.smithi179.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-29T11:17:46.146 INFO:teuthology.orchestra.run.smithi179.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-29T11:17:46.146 INFO:teuthology.orchestra.run.smithi179.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-29T11:17:46.146 INFO:teuthology.orchestra.run.smithi179.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-29T11:17:46.146 INFO:teuthology.orchestra.run.smithi179.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-29T11:17:46.146 INFO:teuthology.orchestra.run.smithi179.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-29T11:17:46.146 INFO:teuthology.orchestra.run.smithi179.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-29T11:17:46.146 INFO:teuthology.orchestra.run.smithi179.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-29T11:17:46.146 INFO:teuthology.orchestra.run.smithi179.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-29T11:17:46.146 INFO:teuthology.orchestra.run.smithi179.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-29T11:17:46.146 INFO:teuthology.orchestra.run.smithi179.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-29T11:17:46.146 INFO:teuthology.orchestra.run.smithi179.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-29T11:17:46.146 INFO:teuthology.orchestra.run.smithi179.stdout: python3-ply-3.11-14.el9.noarch 2024-03-29T11:17:46.147 INFO:teuthology.orchestra.run.smithi179.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-29T11:17:46.147 INFO:teuthology.orchestra.run.smithi179.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-29T11:17:46.147 INFO:teuthology.orchestra.run.smithi179.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-29T11:17:46.147 INFO:teuthology.orchestra.run.smithi179.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-29T11:17:46.147 INFO:teuthology.orchestra.run.smithi179.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-29T11:17:46.147 INFO:teuthology.orchestra.run.smithi179.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-29T11:17:46.147 INFO:teuthology.orchestra.run.smithi179.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-29T11:17:46.147 INFO:teuthology.orchestra.run.smithi179.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-29T11:17:46.147 INFO:teuthology.orchestra.run.smithi179.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-29T11:17:46.147 INFO:teuthology.orchestra.run.smithi179.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-29T11:17:46.147 INFO:teuthology.orchestra.run.smithi179.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-29T11:17:46.147 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-29T11:17:46.147 INFO:teuthology.orchestra.run.smithi179.stdout:Complete! 2024-03-29T11:17:46.163 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-ply-3.11-14.el9.noarch 13/36 2024-03-29T11:17:46.219 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 14/36 2024-03-29T11:17:46.435 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 15/36 2024-03-29T11:17:46.484 DEBUG:teuthology.orchestra.run.smithi179:> sudo yum -y install ceph-base 2024-03-29T11:17:46.485 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 16/36 2024-03-29T11:17:46.524 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 17/36 2024-03-29T11:17:46.558 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 18/36 2024-03-29T11:17:46.588 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 19/36 2024-03-29T11:17:46.663 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 20/36 2024-03-29T11:17:46.698 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-29T11:17:46.732 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 22/36 2024-03-29T11:17:46.772 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 23/36 2024-03-29T11:17:46.807 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 24/36 2024-03-29T11:17:46.841 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 25/36 2024-03-29T11:17:46.930 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 26/36 2024-03-29T11:17:46.964 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 27/36 2024-03-29T11:17:47.035 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 28/36 2024-03-29T11:17:47.061 INFO:teuthology.orchestra.run.smithi179.stdout:Last metadata expiration check: 0:01:30 ago on Fri 29 Mar 2024 11:16:17 AM UTC. 2024-03-29T11:17:47.159 INFO:teuthology.orchestra.run.smithi179.stdout:Package ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:17:47.172 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 29/36 2024-03-29T11:17:47.212 INFO:teuthology.orchestra.run.smithi179.stdout:Dependencies resolved. 2024-03-29T11:17:47.213 INFO:teuthology.orchestra.run.smithi179.stdout:Nothing to do. 2024-03-29T11:17:47.213 INFO:teuthology.orchestra.run.smithi179.stdout:Complete! 2024-03-29T11:17:47.268 DEBUG:teuthology.orchestra.run.smithi179:> sudo yum -y install cephadm 2024-03-29T11:17:47.298 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-29T11:17:47.463 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9. 31/36 2024-03-29T11:17:47.502 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/36 2024-03-29T11:17:47.530 INFO:teuthology.orchestra.run.smithi084.stdout: Running scriptlet: ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/36 2024-03-29T11:17:47.530 INFO:teuthology.orchestra.run.smithi084.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T11:17:47.530 INFO:teuthology.orchestra.run.smithi084.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-29T11:17:47.530 INFO:teuthology.orchestra.run.smithi084.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-29T11:17:47.530 INFO:teuthology.orchestra.run.smithi084.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-29T11:17:47.530 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-03-29T11:17:47.833 INFO:teuthology.orchestra.run.smithi179.stdout:Last metadata expiration check: 0:01:30 ago on Fri 29 Mar 2024 11:16:17 AM UTC. 2024-03-29T11:17:47.984 INFO:teuthology.orchestra.run.smithi179.stdout:Dependencies resolved. 2024-03-29T11:17:47.984 INFO:teuthology.orchestra.run.smithi179.stdout:================================================================================ 2024-03-29T11:17:47.985 INFO:teuthology.orchestra.run.smithi179.stdout: Package Arch Version Repository Size 2024-03-29T11:17:47.985 INFO:teuthology.orchestra.run.smithi179.stdout:================================================================================ 2024-03-29T11:17:47.985 INFO:teuthology.orchestra.run.smithi179.stdout:Installing: 2024-03-29T11:17:47.985 INFO:teuthology.orchestra.run.smithi179.stdout: cephadm noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 747 k 2024-03-29T11:17:47.985 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-29T11:17:47.985 INFO:teuthology.orchestra.run.smithi179.stdout:Transaction Summary 2024-03-29T11:17:47.985 INFO:teuthology.orchestra.run.smithi179.stdout:================================================================================ 2024-03-29T11:17:47.985 INFO:teuthology.orchestra.run.smithi179.stdout:Install 1 Package 2024-03-29T11:17:47.985 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-29T11:17:47.985 INFO:teuthology.orchestra.run.smithi179.stdout:Total download size: 747 k 2024-03-29T11:17:47.985 INFO:teuthology.orchestra.run.smithi179.stdout:Installed size: 754 k 2024-03-29T11:17:47.985 INFO:teuthology.orchestra.run.smithi179.stdout:Downloading Packages: 2024-03-29T11:17:48.789 INFO:teuthology.orchestra.run.smithi179.stdout:cephadm-19.0.0-1439.g5dcb190d.el9.noarch.rpm 931 kB/s | 747 kB 00:00 2024-03-29T11:17:48.790 INFO:teuthology.orchestra.run.smithi179.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:17:48.790 INFO:teuthology.orchestra.run.smithi179.stdout:Total 929 kB/s | 747 kB 00:00 2024-03-29T11:17:48.790 INFO:teuthology.orchestra.run.smithi179.stdout:Running transaction check 2024-03-29T11:17:48.795 INFO:teuthology.orchestra.run.smithi179.stdout:Transaction check succeeded. 2024-03-29T11:17:48.795 INFO:teuthology.orchestra.run.smithi179.stdout:Running transaction test 2024-03-29T11:17:48.801 INFO:teuthology.orchestra.run.smithi179.stdout:Transaction test succeeded. 2024-03-29T11:17:48.801 INFO:teuthology.orchestra.run.smithi179.stdout:Running transaction 2024-03-29T11:17:48.874 INFO:teuthology.orchestra.run.smithi179.stdout: Preparing : 1/1 2024-03-29T11:17:48.961 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/36 2024-03-29T11:17:48.987 INFO:teuthology.orchestra.run.smithi084.stdout: Running scriptlet: ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/36 2024-03-29T11:17:48.987 INFO:teuthology.orchestra.run.smithi084.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T11:17:48.987 INFO:teuthology.orchestra.run.smithi084.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-29T11:17:48.987 INFO:teuthology.orchestra.run.smithi084.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-29T11:17:48.987 INFO:teuthology.orchestra.run.smithi084.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-29T11:17:48.987 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-03-29T11:17:49.356 INFO:teuthology.orchestra.run.smithi179.stdout: Running scriptlet: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-29T11:17:49.428 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 34/36 2024-03-29T11:17:49.443 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-29T11:17:49.451 INFO:teuthology.orchestra.run.smithi084.stdout: Running scriptlet: ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 34/36 2024-03-29T11:17:49.451 INFO:teuthology.orchestra.run.smithi084.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T11:17:49.451 INFO:teuthology.orchestra.run.smithi084.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-29T11:17:49.451 INFO:teuthology.orchestra.run.smithi084.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-29T11:17:49.451 INFO:teuthology.orchestra.run.smithi084.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-29T11:17:49.451 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-03-29T11:17:49.662 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 35/36 2024-03-29T11:17:49.686 INFO:teuthology.orchestra.run.smithi084.stdout: Running scriptlet: ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 35/36 2024-03-29T11:17:49.686 INFO:teuthology.orchestra.run.smithi084.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T11:17:49.686 INFO:teuthology.orchestra.run.smithi084.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-29T11:17:49.686 INFO:teuthology.orchestra.run.smithi084.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-29T11:17:49.686 INFO:teuthology.orchestra.run.smithi084.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-29T11:17:49.686 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-03-29T11:17:49.751 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 36/36 2024-03-29T11:17:49.859 INFO:teuthology.orchestra.run.smithi179.stdout: Running scriptlet: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-29T11:17:50.133 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-29T11:17:50.133 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-29T11:17:50.133 INFO:teuthology.orchestra.run.smithi179.stdout:Installed: 2024-03-29T11:17:50.133 INFO:teuthology.orchestra.run.smithi179.stdout: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:17:50.133 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-29T11:17:50.133 INFO:teuthology.orchestra.run.smithi179.stdout:Complete! 2024-03-29T11:17:50.380 DEBUG:teuthology.orchestra.run.smithi179:> sudo yum -y install ceph-immutable-object-cache 2024-03-29T11:17:50.952 INFO:teuthology.orchestra.run.smithi179.stdout:Last metadata expiration check: 0:01:33 ago on Fri 29 Mar 2024 11:16:17 AM UTC. 2024-03-29T11:17:51.106 INFO:teuthology.orchestra.run.smithi179.stdout:Dependencies resolved. 2024-03-29T11:17:51.107 INFO:teuthology.orchestra.run.smithi179.stdout:================================================================================ 2024-03-29T11:17:51.107 INFO:teuthology.orchestra.run.smithi179.stdout: Package Arch Version Repo Size 2024-03-29T11:17:51.107 INFO:teuthology.orchestra.run.smithi179.stdout:================================================================================ 2024-03-29T11:17:51.107 INFO:teuthology.orchestra.run.smithi179.stdout:Installing: 2024-03-29T11:17:51.107 INFO:teuthology.orchestra.run.smithi179.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 145 k 2024-03-29T11:17:51.108 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-29T11:17:51.108 INFO:teuthology.orchestra.run.smithi179.stdout:Transaction Summary 2024-03-29T11:17:51.108 INFO:teuthology.orchestra.run.smithi179.stdout:================================================================================ 2024-03-29T11:17:51.108 INFO:teuthology.orchestra.run.smithi179.stdout:Install 1 Package 2024-03-29T11:17:51.108 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-29T11:17:51.108 INFO:teuthology.orchestra.run.smithi179.stdout:Total download size: 145 k 2024-03-29T11:17:51.108 INFO:teuthology.orchestra.run.smithi179.stdout:Installed size: 436 k 2024-03-29T11:17:51.109 INFO:teuthology.orchestra.run.smithi179.stdout:Downloading Packages: 2024-03-29T11:17:51.470 INFO:teuthology.orchestra.run.smithi084.stdout: Running scriptlet: ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 36/36 2024-03-29T11:17:51.470 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/36 2024-03-29T11:17:51.470 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/36 2024-03-29T11:17:51.470 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 3/36 2024-03-29T11:17:51.471 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 4/36 2024-03-29T11:17:51.471 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/36 2024-03-29T11:17:51.471 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9. 6/36 2024-03-29T11:17:51.471 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : lua-5.4.4-4.el9.x86_64 7/36 2024-03-29T11:17:51.471 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 8/36 2024-03-29T11:17:51.471 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 9/36 2024-03-29T11:17:51.471 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 10/36 2024-03-29T11:17:51.472 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 11/36 2024-03-29T11:17:51.472 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 12/36 2024-03-29T11:17:51.472 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 13/36 2024-03-29T11:17:51.472 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-ply-3.11-14.el9.noarch 14/36 2024-03-29T11:17:51.472 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 15/36 2024-03-29T11:17:51.472 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 16/36 2024-03-29T11:17:51.472 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 17/36 2024-03-29T11:17:51.472 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 18/36 2024-03-29T11:17:51.473 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 19/36 2024-03-29T11:17:51.473 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 20/36 2024-03-29T11:17:51.473 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-29T11:17:51.473 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 22/36 2024-03-29T11:17:51.473 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 23/36 2024-03-29T11:17:51.473 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 24/36 2024-03-29T11:17:51.473 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 25/36 2024-03-29T11:17:51.473 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 26/36 2024-03-29T11:17:51.473 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 27/36 2024-03-29T11:17:51.473 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 28/36 2024-03-29T11:17:51.473 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 29/36 2024-03-29T11:17:51.473 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-29T11:17:51.474 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 31/36 2024-03-29T11:17:51.474 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 32/36 2024-03-29T11:17:51.474 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 33/36 2024-03-29T11:17:51.474 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 34/36 2024-03-29T11:17:51.474 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 35/36 2024-03-29T11:17:51.503 INFO:teuthology.orchestra.run.smithi179.stdout:ceph-immutable-object-cache-19.0.0-1439.g5dcb19 367 kB/s | 145 kB 00:00 2024-03-29T11:17:51.504 INFO:teuthology.orchestra.run.smithi179.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:17:51.504 INFO:teuthology.orchestra.run.smithi179.stdout:Total 366 kB/s | 145 kB 00:00 2024-03-29T11:17:51.504 INFO:teuthology.orchestra.run.smithi179.stdout:Running transaction check 2024-03-29T11:17:51.512 INFO:teuthology.orchestra.run.smithi179.stdout:Transaction check succeeded. 2024-03-29T11:17:51.512 INFO:teuthology.orchestra.run.smithi179.stdout:Running transaction test 2024-03-29T11:17:51.565 INFO:teuthology.orchestra.run.smithi179.stdout:Transaction test succeeded. 2024-03-29T11:17:51.565 INFO:teuthology.orchestra.run.smithi179.stdout:Running transaction 2024-03-29T11:17:51.705 INFO:teuthology.orchestra.run.smithi179.stdout: Preparing : 1/1 2024-03-29T11:17:51.771 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-29T11:17:51.796 INFO:teuthology.orchestra.run.smithi179.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-29T11:17:51.796 INFO:teuthology.orchestra.run.smithi179.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T11:17:51.796 INFO:teuthology.orchestra.run.smithi179.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-29T11:17:51.796 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-29T11:17:52.348 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 36/36 2024-03-29T11:17:52.348 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-03-29T11:17:52.348 INFO:teuthology.orchestra.run.smithi084.stdout:Installed: 2024-03-29T11:17:52.348 INFO:teuthology.orchestra.run.smithi084.stdout: ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:17:52.349 INFO:teuthology.orchestra.run.smithi084.stdout: ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:17:52.349 INFO:teuthology.orchestra.run.smithi084.stdout: ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:17:52.349 INFO:teuthology.orchestra.run.smithi084.stdout: ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:17:52.349 INFO:teuthology.orchestra.run.smithi084.stdout: ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:17:52.349 INFO:teuthology.orchestra.run.smithi084.stdout: ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:17:52.349 INFO:teuthology.orchestra.run.smithi084.stdout: lua-5.4.4-4.el9.x86_64 2024-03-29T11:17:52.349 INFO:teuthology.orchestra.run.smithi084.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-03-29T11:17:52.349 INFO:teuthology.orchestra.run.smithi084.stdout: luarocks-3.9.2-1.el9.noarch 2024-03-29T11:17:52.349 INFO:teuthology.orchestra.run.smithi084.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-29T11:17:52.349 INFO:teuthology.orchestra.run.smithi084.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-29T11:17:52.349 INFO:teuthology.orchestra.run.smithi084.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-29T11:17:52.349 INFO:teuthology.orchestra.run.smithi084.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-29T11:17:52.350 INFO:teuthology.orchestra.run.smithi084.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-29T11:17:52.350 INFO:teuthology.orchestra.run.smithi084.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-29T11:17:52.350 INFO:teuthology.orchestra.run.smithi084.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-29T11:17:52.350 INFO:teuthology.orchestra.run.smithi084.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-29T11:17:52.350 INFO:teuthology.orchestra.run.smithi084.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-29T11:17:52.350 INFO:teuthology.orchestra.run.smithi084.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-29T11:17:52.350 INFO:teuthology.orchestra.run.smithi084.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-29T11:17:52.350 INFO:teuthology.orchestra.run.smithi084.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-29T11:17:52.350 INFO:teuthology.orchestra.run.smithi084.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-29T11:17:52.350 INFO:teuthology.orchestra.run.smithi084.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-29T11:17:52.350 INFO:teuthology.orchestra.run.smithi084.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-29T11:17:52.350 INFO:teuthology.orchestra.run.smithi084.stdout: python3-ply-3.11-14.el9.noarch 2024-03-29T11:17:52.350 INFO:teuthology.orchestra.run.smithi084.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-29T11:17:52.350 INFO:teuthology.orchestra.run.smithi084.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-29T11:17:52.351 INFO:teuthology.orchestra.run.smithi084.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-29T11:17:52.351 INFO:teuthology.orchestra.run.smithi084.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-29T11:17:52.351 INFO:teuthology.orchestra.run.smithi084.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-29T11:17:52.351 INFO:teuthology.orchestra.run.smithi084.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-29T11:17:52.351 INFO:teuthology.orchestra.run.smithi084.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-29T11:17:52.351 INFO:teuthology.orchestra.run.smithi084.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-29T11:17:52.351 INFO:teuthology.orchestra.run.smithi084.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-29T11:17:52.351 INFO:teuthology.orchestra.run.smithi084.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-29T11:17:52.351 INFO:teuthology.orchestra.run.smithi084.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-29T11:17:52.351 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-03-29T11:17:52.351 INFO:teuthology.orchestra.run.smithi084.stdout:Complete! 2024-03-29T11:17:52.730 DEBUG:teuthology.orchestra.run.smithi084:> sudo yum -y install ceph-base 2024-03-29T11:17:52.786 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-29T11:17:52.786 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-29T11:17:52.786 INFO:teuthology.orchestra.run.smithi179.stdout:Installed: 2024-03-29T11:17:52.786 INFO:teuthology.orchestra.run.smithi179.stdout: ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:17:52.786 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-29T11:17:52.787 INFO:teuthology.orchestra.run.smithi179.stdout:Complete! 2024-03-29T11:17:53.032 DEBUG:teuthology.orchestra.run.smithi179:> sudo yum -y install ceph-mgr 2024-03-29T11:17:53.300 INFO:teuthology.orchestra.run.smithi084.stdout:Last metadata expiration check: 0:01:41 ago on Fri 29 Mar 2024 11:16:12 AM UTC. 2024-03-29T11:17:53.398 INFO:teuthology.orchestra.run.smithi084.stdout:Package ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:17:53.449 INFO:teuthology.orchestra.run.smithi084.stdout:Dependencies resolved. 2024-03-29T11:17:53.450 INFO:teuthology.orchestra.run.smithi084.stdout:Nothing to do. 2024-03-29T11:17:53.450 INFO:teuthology.orchestra.run.smithi084.stdout:Complete! 2024-03-29T11:17:53.508 DEBUG:teuthology.orchestra.run.smithi084:> sudo yum -y install cephadm 2024-03-29T11:17:53.616 INFO:teuthology.orchestra.run.smithi179.stdout:Last metadata expiration check: 0:01:36 ago on Fri 29 Mar 2024 11:16:17 AM UTC. 2024-03-29T11:17:53.716 INFO:teuthology.orchestra.run.smithi179.stdout:Package ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:17:53.769 INFO:teuthology.orchestra.run.smithi179.stdout:Dependencies resolved. 2024-03-29T11:17:53.770 INFO:teuthology.orchestra.run.smithi179.stdout:Nothing to do. 2024-03-29T11:17:53.770 INFO:teuthology.orchestra.run.smithi179.stdout:Complete! 2024-03-29T11:17:53.825 DEBUG:teuthology.orchestra.run.smithi179:> sudo yum -y install ceph-mgr-dashboard 2024-03-29T11:17:54.067 INFO:teuthology.orchestra.run.smithi084.stdout:Last metadata expiration check: 0:01:42 ago on Fri 29 Mar 2024 11:16:12 AM UTC. 2024-03-29T11:17:54.219 INFO:teuthology.orchestra.run.smithi084.stdout:Dependencies resolved. 2024-03-29T11:17:54.220 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2024-03-29T11:17:54.220 INFO:teuthology.orchestra.run.smithi084.stdout: Package Arch Version Repository Size 2024-03-29T11:17:54.220 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2024-03-29T11:17:54.220 INFO:teuthology.orchestra.run.smithi084.stdout:Installing: 2024-03-29T11:17:54.220 INFO:teuthology.orchestra.run.smithi084.stdout: cephadm noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 747 k 2024-03-29T11:17:54.220 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-03-29T11:17:54.220 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction Summary 2024-03-29T11:17:54.220 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2024-03-29T11:17:54.220 INFO:teuthology.orchestra.run.smithi084.stdout:Install 1 Package 2024-03-29T11:17:54.220 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-03-29T11:17:54.220 INFO:teuthology.orchestra.run.smithi084.stdout:Total download size: 747 k 2024-03-29T11:17:54.220 INFO:teuthology.orchestra.run.smithi084.stdout:Installed size: 754 k 2024-03-29T11:17:54.221 INFO:teuthology.orchestra.run.smithi084.stdout:Downloading Packages: 2024-03-29T11:17:54.384 INFO:teuthology.orchestra.run.smithi179.stdout:Last metadata expiration check: 0:01:37 ago on Fri 29 Mar 2024 11:16:17 AM UTC. 2024-03-29T11:17:54.536 INFO:teuthology.orchestra.run.smithi179.stdout:Dependencies resolved. 2024-03-29T11:17:54.537 INFO:teuthology.orchestra.run.smithi179.stdout:====================================================================================== 2024-03-29T11:17:54.538 INFO:teuthology.orchestra.run.smithi179.stdout: Package Arch Version Repository Size 2024-03-29T11:17:54.538 INFO:teuthology.orchestra.run.smithi179.stdout:====================================================================================== 2024-03-29T11:17:54.538 INFO:teuthology.orchestra.run.smithi179.stdout:Installing: 2024-03-29T11:17:54.538 INFO:teuthology.orchestra.run.smithi179.stdout: ceph-mgr-dashboard noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 1.8 M 2024-03-29T11:17:54.538 INFO:teuthology.orchestra.run.smithi179.stdout:Installing dependencies: 2024-03-29T11:17:54.538 INFO:teuthology.orchestra.run.smithi179.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-03-29T11:17:54.538 INFO:teuthology.orchestra.run.smithi179.stdout: ceph-grafana-dashboards noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 31 k 2024-03-29T11:17:54.538 INFO:teuthology.orchestra.run.smithi179.stdout: ceph-prometheus-alerts noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 15 k 2024-03-29T11:17:54.538 INFO:teuthology.orchestra.run.smithi179.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-03-29T11:17:54.538 INFO:teuthology.orchestra.run.smithi179.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-03-29T11:17:54.538 INFO:teuthology.orchestra.run.smithi179.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-03-29T11:17:54.539 INFO:teuthology.orchestra.run.smithi179.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-03-29T11:17:54.539 INFO:teuthology.orchestra.run.smithi179.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-03-29T11:17:54.539 INFO:teuthology.orchestra.run.smithi179.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-03-29T11:17:54.539 INFO:teuthology.orchestra.run.smithi179.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-29T11:17:54.539 INFO:teuthology.orchestra.run.smithi179.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-29T11:17:54.539 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-29T11:17:54.539 INFO:teuthology.orchestra.run.smithi179.stdout:Transaction Summary 2024-03-29T11:17:54.539 INFO:teuthology.orchestra.run.smithi179.stdout:====================================================================================== 2024-03-29T11:17:54.539 INFO:teuthology.orchestra.run.smithi179.stdout:Install 12 Packages 2024-03-29T11:17:54.539 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-29T11:17:54.540 INFO:teuthology.orchestra.run.smithi179.stdout:Total download size: 6.9 M 2024-03-29T11:17:54.540 INFO:teuthology.orchestra.run.smithi179.stdout:Installed size: 26 M 2024-03-29T11:17:54.540 INFO:teuthology.orchestra.run.smithi179.stdout:Downloading Packages: 2024-03-29T11:17:54.566 INFO:teuthology.orchestra.run.smithi084.stdout:cephadm-19.0.0-1439.g5dcb190d.el9.noarch.rpm 2.1 MB/s | 747 kB 00:00 2024-03-29T11:17:54.567 INFO:teuthology.orchestra.run.smithi084.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:17:54.567 INFO:teuthology.orchestra.run.smithi084.stdout:Total 2.1 MB/s | 747 kB 00:00 2024-03-29T11:17:54.567 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction check 2024-03-29T11:17:54.572 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction check succeeded. 2024-03-29T11:17:54.573 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction test 2024-03-29T11:17:54.578 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction test succeeded. 2024-03-29T11:17:54.578 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction 2024-03-29T11:17:54.649 INFO:teuthology.orchestra.run.smithi084.stdout: Preparing : 1/1 2024-03-29T11:17:55.075 INFO:teuthology.orchestra.run.smithi179.stdout:(1/12): ceph-grafana-dashboards-19.0.0-1439.g5d 114 kB/s | 31 kB 00:00 2024-03-29T11:17:55.099 INFO:teuthology.orchestra.run.smithi084.stdout: Running scriptlet: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-29T11:17:55.125 INFO:teuthology.orchestra.run.smithi179.stdout:(2/12): ceph-prometheus-alerts-19.0.0-1439.g5dc 47 kB/s | 15 kB 00:00 2024-03-29T11:17:55.179 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-29T11:17:55.217 INFO:teuthology.orchestra.run.smithi179.stdout:(3/12): protobuf-3.14.0-13.el9.x86_64.rpm 7.1 MB/s | 1.0 MB 00:00 2024-03-29T11:17:55.243 INFO:teuthology.orchestra.run.smithi179.stdout:(4/12): python3-protobuf-3.14.0-13.el9.noarch.r 2.0 MB/s | 238 kB 00:00 2024-03-29T11:17:55.476 INFO:teuthology.orchestra.run.smithi179.stdout:(5/12): abseil-cpp-20211102.0-3.el9.x86_64.rpm 2.3 MB/s | 548 kB 00:00 2024-03-29T11:17:55.552 INFO:teuthology.orchestra.run.smithi179.stdout:(6/12): grpc-data-1.46.7-10.el9.noarch.rpm 388 kB/s | 19 kB 00:00 2024-03-29T11:17:55.593 INFO:teuthology.orchestra.run.smithi179.stdout:(7/12): ceph-mgr-dashboard-19.0.0-1439.g5dcb190 2.2 MB/s | 1.8 MB 00:00 2024-03-29T11:17:55.625 INFO:teuthology.orchestra.run.smithi084.stdout: Running scriptlet: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-29T11:17:55.685 INFO:teuthology.orchestra.run.smithi179.stdout:(8/12): python3-grpcio-1.46.7-10.el9.x86_64.rpm 15 MB/s | 2.0 MB 00:00 2024-03-29T11:17:55.702 INFO:teuthology.orchestra.run.smithi179.stdout:(9/12): python3-grpcio-tools-1.46.7-10.el9.x86_ 1.3 MB/s | 144 kB 00:00 2024-03-29T11:17:55.729 INFO:teuthology.orchestra.run.smithi179.stdout:(10/12): python3-repoze-lru-0.7-16.el9.noarch.r 736 kB/s | 31 kB 00:00 2024-03-29T11:17:55.753 INFO:teuthology.orchestra.run.smithi179.stdout:(11/12): python3-routes-2.5.1-5.el9.noarch.rpm 3.7 MB/s | 188 kB 00:00 2024-03-29T11:17:55.972 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-29T11:17:55.973 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-03-29T11:17:55.973 INFO:teuthology.orchestra.run.smithi084.stdout:Installed: 2024-03-29T11:17:55.973 INFO:teuthology.orchestra.run.smithi084.stdout: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:17:55.973 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-03-29T11:17:55.973 INFO:teuthology.orchestra.run.smithi084.stdout:Complete! 2024-03-29T11:17:56.173 DEBUG:teuthology.orchestra.run.smithi084:> sudo yum -y install ceph-immutable-object-cache 2024-03-29T11:17:56.757 INFO:teuthology.orchestra.run.smithi084.stdout:Last metadata expiration check: 0:01:44 ago on Fri 29 Mar 2024 11:16:12 AM UTC. 2024-03-29T11:17:56.907 INFO:teuthology.orchestra.run.smithi084.stdout:Dependencies resolved. 2024-03-29T11:17:56.907 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2024-03-29T11:17:56.907 INFO:teuthology.orchestra.run.smithi084.stdout: Package Arch Version Repo Size 2024-03-29T11:17:56.907 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2024-03-29T11:17:56.908 INFO:teuthology.orchestra.run.smithi084.stdout:Installing: 2024-03-29T11:17:56.908 INFO:teuthology.orchestra.run.smithi084.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 145 k 2024-03-29T11:17:56.908 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-03-29T11:17:56.908 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction Summary 2024-03-29T11:17:56.908 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2024-03-29T11:17:56.908 INFO:teuthology.orchestra.run.smithi084.stdout:Install 1 Package 2024-03-29T11:17:56.908 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-03-29T11:17:56.908 INFO:teuthology.orchestra.run.smithi084.stdout:Total download size: 145 k 2024-03-29T11:17:56.909 INFO:teuthology.orchestra.run.smithi084.stdout:Installed size: 436 k 2024-03-29T11:17:56.909 INFO:teuthology.orchestra.run.smithi084.stdout:Downloading Packages: 2024-03-29T11:17:56.953 INFO:teuthology.orchestra.run.smithi179.stdout:(12/12): protobuf-compiler-3.14.0-13.el9.x86_64 497 kB/s | 863 kB 00:01 2024-03-29T11:17:56.958 INFO:teuthology.orchestra.run.smithi179.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:17:56.958 INFO:teuthology.orchestra.run.smithi179.stdout:Total 2.8 MB/s | 6.9 MB 00:02 2024-03-29T11:17:56.977 INFO:teuthology.orchestra.run.smithi179.stdout:Running transaction check 2024-03-29T11:17:56.994 INFO:teuthology.orchestra.run.smithi179.stdout:Transaction check succeeded. 2024-03-29T11:17:56.994 INFO:teuthology.orchestra.run.smithi179.stdout:Running transaction test 2024-03-29T11:17:57.172 INFO:teuthology.orchestra.run.smithi179.stdout:Transaction test succeeded. 2024-03-29T11:17:57.173 INFO:teuthology.orchestra.run.smithi179.stdout:Running transaction 2024-03-29T11:17:57.255 INFO:teuthology.orchestra.run.smithi084.stdout:ceph-immutable-object-cache-19.0.0-1439.g5dcb19 420 kB/s | 145 kB 00:00 2024-03-29T11:17:57.256 INFO:teuthology.orchestra.run.smithi084.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:17:57.256 INFO:teuthology.orchestra.run.smithi084.stdout:Total 418 kB/s | 145 kB 00:00 2024-03-29T11:17:57.256 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction check 2024-03-29T11:17:57.263 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction check succeeded. 2024-03-29T11:17:57.264 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction test 2024-03-29T11:17:57.315 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction test succeeded. 2024-03-29T11:17:57.316 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction 2024-03-29T11:17:57.466 INFO:teuthology.orchestra.run.smithi084.stdout: Preparing : 1/1 2024-03-29T11:17:57.515 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-29T11:17:57.540 INFO:teuthology.orchestra.run.smithi084.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-29T11:17:57.540 INFO:teuthology.orchestra.run.smithi084.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T11:17:57.540 INFO:teuthology.orchestra.run.smithi084.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-29T11:17:57.540 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-03-29T11:17:57.744 INFO:teuthology.orchestra.run.smithi179.stdout: Preparing : 1/1 2024-03-29T11:17:57.821 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 1/12 2024-03-29T11:17:57.864 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 2/12 2024-03-29T11:17:57.910 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/12 2024-03-29T11:17:57.960 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/12 2024-03-29T11:17:58.077 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 5/12 2024-03-29T11:17:58.201 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 6/12 2024-03-29T11:17:58.311 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 7/12 2024-03-29T11:17:58.354 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 8/12 2024-03-29T11:17:58.382 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/12 2024-03-29T11:17:58.414 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9 10/12 2024-03-29T11:17:58.547 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-29T11:17:58.547 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-03-29T11:17:58.547 INFO:teuthology.orchestra.run.smithi084.stdout:Installed: 2024-03-29T11:17:58.547 INFO:teuthology.orchestra.run.smithi084.stdout: ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:17:58.547 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-03-29T11:17:58.547 INFO:teuthology.orchestra.run.smithi084.stdout:Complete! 2024-03-29T11:17:58.656 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el 11/12 2024-03-29T11:17:58.690 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 12/12 2024-03-29T11:17:58.783 DEBUG:teuthology.orchestra.run.smithi084:> sudo yum -y install ceph-mgr 2024-03-29T11:17:59.359 INFO:teuthology.orchestra.run.smithi084.stdout:Last metadata expiration check: 0:01:47 ago on Fri 29 Mar 2024 11:16:12 AM UTC. 2024-03-29T11:17:59.459 INFO:teuthology.orchestra.run.smithi084.stdout:Package ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:17:59.507 INFO:teuthology.orchestra.run.smithi179.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 12/12 2024-03-29T11:17:59.507 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el 1/12 2024-03-29T11:17:59.507 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 2/12 2024-03-29T11:17:59.507 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9 3/12 2024-03-29T11:17:59.507 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 4/12 2024-03-29T11:17:59.507 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 5/12 2024-03-29T11:17:59.507 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 6/12 2024-03-29T11:17:59.507 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 7/12 2024-03-29T11:17:59.508 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 8/12 2024-03-29T11:17:59.508 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 9/12 2024-03-29T11:17:59.508 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 10/12 2024-03-29T11:17:59.508 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 11/12 2024-03-29T11:17:59.512 INFO:teuthology.orchestra.run.smithi084.stdout:Dependencies resolved. 2024-03-29T11:17:59.514 INFO:teuthology.orchestra.run.smithi084.stdout:Nothing to do. 2024-03-29T11:17:59.514 INFO:teuthology.orchestra.run.smithi084.stdout:Complete! 2024-03-29T11:17:59.574 DEBUG:teuthology.orchestra.run.smithi084:> sudo yum -y install ceph-mgr-dashboard 2024-03-29T11:17:59.851 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 12/12 2024-03-29T11:17:59.851 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-29T11:17:59.851 INFO:teuthology.orchestra.run.smithi179.stdout:Installed: 2024-03-29T11:17:59.851 INFO:teuthology.orchestra.run.smithi179.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-03-29T11:17:59.851 INFO:teuthology.orchestra.run.smithi179.stdout: ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:17:59.851 INFO:teuthology.orchestra.run.smithi179.stdout: ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:17:59.852 INFO:teuthology.orchestra.run.smithi179.stdout: ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:17:59.852 INFO:teuthology.orchestra.run.smithi179.stdout: grpc-data-1.46.7-10.el9.noarch 2024-03-29T11:17:59.852 INFO:teuthology.orchestra.run.smithi179.stdout: protobuf-3.14.0-13.el9.x86_64 2024-03-29T11:17:59.852 INFO:teuthology.orchestra.run.smithi179.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-03-29T11:17:59.852 INFO:teuthology.orchestra.run.smithi179.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-03-29T11:17:59.852 INFO:teuthology.orchestra.run.smithi179.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-03-29T11:17:59.852 INFO:teuthology.orchestra.run.smithi179.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-03-29T11:17:59.852 INFO:teuthology.orchestra.run.smithi179.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-29T11:17:59.852 INFO:teuthology.orchestra.run.smithi179.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-29T11:17:59.852 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-29T11:17:59.852 INFO:teuthology.orchestra.run.smithi179.stdout:Complete! 2024-03-29T11:18:00.075 DEBUG:teuthology.orchestra.run.smithi179:> sudo yum -y install ceph-mgr-diskprediction-local 2024-03-29T11:18:00.137 INFO:teuthology.orchestra.run.smithi084.stdout:Last metadata expiration check: 0:01:48 ago on Fri 29 Mar 2024 11:16:12 AM UTC. 2024-03-29T11:18:00.288 INFO:teuthology.orchestra.run.smithi084.stdout:Dependencies resolved. 2024-03-29T11:18:00.289 INFO:teuthology.orchestra.run.smithi084.stdout:====================================================================================== 2024-03-29T11:18:00.289 INFO:teuthology.orchestra.run.smithi084.stdout: Package Arch Version Repository Size 2024-03-29T11:18:00.289 INFO:teuthology.orchestra.run.smithi084.stdout:====================================================================================== 2024-03-29T11:18:00.290 INFO:teuthology.orchestra.run.smithi084.stdout:Installing: 2024-03-29T11:18:00.290 INFO:teuthology.orchestra.run.smithi084.stdout: ceph-mgr-dashboard noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 1.8 M 2024-03-29T11:18:00.290 INFO:teuthology.orchestra.run.smithi084.stdout:Installing dependencies: 2024-03-29T11:18:00.290 INFO:teuthology.orchestra.run.smithi084.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-03-29T11:18:00.290 INFO:teuthology.orchestra.run.smithi084.stdout: ceph-grafana-dashboards noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 31 k 2024-03-29T11:18:00.290 INFO:teuthology.orchestra.run.smithi084.stdout: ceph-prometheus-alerts noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 15 k 2024-03-29T11:18:00.290 INFO:teuthology.orchestra.run.smithi084.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-03-29T11:18:00.290 INFO:teuthology.orchestra.run.smithi084.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-03-29T11:18:00.290 INFO:teuthology.orchestra.run.smithi084.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-03-29T11:18:00.290 INFO:teuthology.orchestra.run.smithi084.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-03-29T11:18:00.290 INFO:teuthology.orchestra.run.smithi084.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-03-29T11:18:00.290 INFO:teuthology.orchestra.run.smithi084.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-03-29T11:18:00.290 INFO:teuthology.orchestra.run.smithi084.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-29T11:18:00.291 INFO:teuthology.orchestra.run.smithi084.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-29T11:18:00.291 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-03-29T11:18:00.291 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction Summary 2024-03-29T11:18:00.291 INFO:teuthology.orchestra.run.smithi084.stdout:====================================================================================== 2024-03-29T11:18:00.291 INFO:teuthology.orchestra.run.smithi084.stdout:Install 12 Packages 2024-03-29T11:18:00.291 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-03-29T11:18:00.291 INFO:teuthology.orchestra.run.smithi084.stdout:Total download size: 6.9 M 2024-03-29T11:18:00.292 INFO:teuthology.orchestra.run.smithi084.stdout:Installed size: 26 M 2024-03-29T11:18:00.292 INFO:teuthology.orchestra.run.smithi084.stdout:Downloading Packages: 2024-03-29T11:18:00.643 INFO:teuthology.orchestra.run.smithi179.stdout:Last metadata expiration check: 0:01:43 ago on Fri 29 Mar 2024 11:16:17 AM UTC. 2024-03-29T11:18:00.750 INFO:teuthology.orchestra.run.smithi084.stdout:(1/12): ceph-prometheus-alerts-19.0.0-1439.g5dc 76 kB/s | 15 kB 00:00 2024-03-29T11:18:00.792 INFO:teuthology.orchestra.run.smithi179.stdout:Dependencies resolved. 2024-03-29T11:18:00.793 INFO:teuthology.orchestra.run.smithi179.stdout:============================================================================================ 2024-03-29T11:18:00.793 INFO:teuthology.orchestra.run.smithi179.stdout: Package Arch Version Repository Size 2024-03-29T11:18:00.793 INFO:teuthology.orchestra.run.smithi179.stdout:============================================================================================ 2024-03-29T11:18:00.793 INFO:teuthology.orchestra.run.smithi179.stdout:Installing: 2024-03-29T11:18:00.793 INFO:teuthology.orchestra.run.smithi179.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 7.4 M 2024-03-29T11:18:00.793 INFO:teuthology.orchestra.run.smithi179.stdout:Installing dependencies: 2024-03-29T11:18:00.793 INFO:teuthology.orchestra.run.smithi179.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-29T11:18:00.793 INFO:teuthology.orchestra.run.smithi179.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-29T11:18:00.793 INFO:teuthology.orchestra.run.smithi179.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-29T11:18:00.794 INFO:teuthology.orchestra.run.smithi179.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-03-29T11:18:00.794 INFO:teuthology.orchestra.run.smithi179.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-03-29T11:18:00.794 INFO:teuthology.orchestra.run.smithi179.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-29T11:18:00.794 INFO:teuthology.orchestra.run.smithi179.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-29T11:18:00.794 INFO:teuthology.orchestra.run.smithi179.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-29T11:18:00.794 INFO:teuthology.orchestra.run.smithi179.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-29T11:18:00.794 INFO:teuthology.orchestra.run.smithi179.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-29T11:18:00.794 INFO:teuthology.orchestra.run.smithi179.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-29T11:18:00.794 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-29T11:18:00.794 INFO:teuthology.orchestra.run.smithi179.stdout:Transaction Summary 2024-03-29T11:18:00.795 INFO:teuthology.orchestra.run.smithi179.stdout:============================================================================================ 2024-03-29T11:18:00.795 INFO:teuthology.orchestra.run.smithi179.stdout:Install 12 Packages 2024-03-29T11:18:00.795 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-29T11:18:00.795 INFO:teuthology.orchestra.run.smithi179.stdout:Total download size: 37 M 2024-03-29T11:18:00.795 INFO:teuthology.orchestra.run.smithi179.stdout:Installed size: 209 M 2024-03-29T11:18:00.796 INFO:teuthology.orchestra.run.smithi179.stdout:Downloading Packages: 2024-03-29T11:18:00.866 INFO:teuthology.orchestra.run.smithi084.stdout:(2/12): ceph-grafana-dashboards-19.0.0-1439.g5d 99 kB/s | 31 kB 00:00 2024-03-29T11:18:01.027 INFO:teuthology.orchestra.run.smithi179.stdout:(1/12): flexiblas-3.0.4-8.el9.x86_64.rpm 250 kB/s | 32 kB 00:00 2024-03-29T11:18:01.033 INFO:teuthology.orchestra.run.smithi084.stdout:(3/12): protobuf-3.14.0-13.el9.x86_64.rpm 3.6 MB/s | 1.0 MB 00:00 2024-03-29T11:18:01.058 INFO:teuthology.orchestra.run.smithi084.stdout:(4/12): python3-protobuf-3.14.0-13.el9.noarch.r 1.2 MB/s | 238 kB 00:00 2024-03-29T11:18:01.070 INFO:teuthology.orchestra.run.smithi179.stdout:(2/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 408 kB/s | 17 kB 00:00 2024-03-29T11:18:01.111 INFO:teuthology.orchestra.run.smithi179.stdout:(3/12): openblas-0.3.21-2.el9.x86_64.rpm 845 kB/s | 34 kB 00:00 2024-03-29T11:18:01.117 INFO:teuthology.orchestra.run.smithi084.stdout:(5/12): ceph-mgr-dashboard-19.0.0-1439.g5dcb190 3.1 MB/s | 1.8 MB 00:00 2024-03-29T11:18:01.201 INFO:teuthology.orchestra.run.smithi084.stdout:(6/12): protobuf-compiler-3.14.0-13.el9.x86_64. 5.1 MB/s | 863 kB 00:00 2024-03-29T11:18:01.242 INFO:teuthology.orchestra.run.smithi084.stdout:(7/12): grpc-data-1.46.7-10.el9.noarch.rpm 155 kB/s | 19 kB 00:00 2024-03-29T11:18:01.278 INFO:teuthology.orchestra.run.smithi179.stdout:(4/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 28 MB/s | 4.7 MB 00:00 2024-03-29T11:18:01.385 INFO:teuthology.orchestra.run.smithi084.stdout:(8/12): abseil-cpp-20211102.0-3.el9.x86_64.rpm 1.6 MB/s | 548 kB 00:00 2024-03-29T11:18:01.401 INFO:teuthology.orchestra.run.smithi084.stdout:(9/12): python3-grpcio-tools-1.46.7-10.el9.x86_ 910 kB/s | 144 kB 00:00 2024-03-29T11:18:01.411 INFO:teuthology.orchestra.run.smithi179.stdout:(5/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 36 MB/s | 4.8 MB 00:00 2024-03-29T11:18:01.443 INFO:teuthology.orchestra.run.smithi084.stdout:(10/12): python3-repoze-lru-0.7-16.el9.noarch.r 526 kB/s | 31 kB 00:00 2024-03-29T11:18:01.453 INFO:teuthology.orchestra.run.smithi179.stdout:(6/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 5.2 MB/s | 217 kB 00:00 2024-03-29T11:18:01.512 INFO:teuthology.orchestra.run.smithi179.stdout:(7/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 4.9 MB/s | 3.0 MB 00:00 2024-03-29T11:18:01.518 INFO:teuthology.orchestra.run.smithi084.stdout:(11/12): python3-routes-2.5.1-5.el9.noarch.rpm 1.6 MB/s | 188 kB 00:00 2024-03-29T11:18:01.671 INFO:teuthology.orchestra.run.smithi179.stdout:(8/12): libgfortran-11.4.1-3.el9.x86_64.rpm 5.0 MB/s | 803 kB 00:00 2024-03-29T11:18:01.685 INFO:teuthology.orchestra.run.smithi084.stdout:(12/12): python3-grpcio-1.46.7-10.el9.x86_64.rp 4.2 MB/s | 2.0 MB 00:00 2024-03-29T11:18:01.688 INFO:teuthology.orchestra.run.smithi084.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:18:01.688 INFO:teuthology.orchestra.run.smithi084.stdout:Total 4.9 MB/s | 6.9 MB 00:01 2024-03-29T11:18:01.706 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction check 2024-03-29T11:18:01.722 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction check succeeded. 2024-03-29T11:18:01.723 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction test 2024-03-29T11:18:01.829 INFO:teuthology.orchestra.run.smithi179.stdout:(9/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 42 MB/s | 16 MB 00:00 2024-03-29T11:18:01.887 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction test succeeded. 2024-03-29T11:18:01.888 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction 2024-03-29T11:18:01.896 INFO:teuthology.orchestra.run.smithi179.stdout:(10/12): libquadmath-11.4.1-3.el9.x86_64.rpm 876 kB/s | 195 kB 00:00 2024-03-29T11:18:02.046 INFO:teuthology.orchestra.run.smithi179.stdout:(11/12): python3-devel-3.9.17-2.el9.x86_64.rpm 1.1 MB/s | 245 kB 00:00 2024-03-29T11:18:02.138 INFO:teuthology.orchestra.run.smithi179.stdout:(12/12): ceph-mgr-diskprediction-local-19.0.0-1 6.0 MB/s | 7.4 MB 00:01 2024-03-29T11:18:02.141 INFO:teuthology.orchestra.run.smithi179.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:18:02.141 INFO:teuthology.orchestra.run.smithi179.stdout:Total 28 MB/s | 37 MB 00:01 2024-03-29T11:18:02.182 INFO:teuthology.orchestra.run.smithi179.stdout:Running transaction check 2024-03-29T11:18:02.206 INFO:teuthology.orchestra.run.smithi179.stdout:Transaction check succeeded. 2024-03-29T11:18:02.206 INFO:teuthology.orchestra.run.smithi179.stdout:Running transaction test 2024-03-29T11:18:02.406 INFO:teuthology.orchestra.run.smithi084.stdout: Preparing : 1/1 2024-03-29T11:18:02.478 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 1/12 2024-03-29T11:18:02.521 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 2/12 2024-03-29T11:18:02.568 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/12 2024-03-29T11:18:02.571 INFO:teuthology.orchestra.run.smithi179.stdout:Transaction test succeeded. 2024-03-29T11:18:02.571 INFO:teuthology.orchestra.run.smithi179.stdout:Running transaction 2024-03-29T11:18:02.598 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/12 2024-03-29T11:18:02.704 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 5/12 2024-03-29T11:18:02.833 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 6/12 2024-03-29T11:18:02.927 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 7/12 2024-03-29T11:18:02.978 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 8/12 2024-03-29T11:18:03.006 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/12 2024-03-29T11:18:03.037 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9 10/12 2024-03-29T11:18:03.273 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el 11/12 2024-03-29T11:18:03.279 INFO:teuthology.orchestra.run.smithi179.stdout: Preparing : 1/1 2024-03-29T11:18:03.321 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 12/12 2024-03-29T11:18:03.344 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 1/12 2024-03-29T11:18:03.384 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-03-29T11:18:03.479 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 3/12 2024-03-29T11:18:03.521 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 4/12 2024-03-29T11:18:03.699 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-29T11:18:03.728 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-29T11:18:03.805 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-03-29T11:18:04.123 INFO:teuthology.orchestra.run.smithi084.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 12/12 2024-03-29T11:18:04.123 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el 1/12 2024-03-29T11:18:04.123 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 2/12 2024-03-29T11:18:04.124 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9 3/12 2024-03-29T11:18:04.124 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 4/12 2024-03-29T11:18:04.124 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 5/12 2024-03-29T11:18:04.124 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 6/12 2024-03-29T11:18:04.124 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 7/12 2024-03-29T11:18:04.124 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 8/12 2024-03-29T11:18:04.124 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 9/12 2024-03-29T11:18:04.124 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 10/12 2024-03-29T11:18:04.124 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 11/12 2024-03-29T11:18:04.457 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 12/12 2024-03-29T11:18:04.458 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-03-29T11:18:04.458 INFO:teuthology.orchestra.run.smithi084.stdout:Installed: 2024-03-29T11:18:04.458 INFO:teuthology.orchestra.run.smithi084.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-03-29T11:18:04.458 INFO:teuthology.orchestra.run.smithi084.stdout: ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:18:04.458 INFO:teuthology.orchestra.run.smithi084.stdout: ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:18:04.458 INFO:teuthology.orchestra.run.smithi084.stdout: ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:18:04.458 INFO:teuthology.orchestra.run.smithi084.stdout: grpc-data-1.46.7-10.el9.noarch 2024-03-29T11:18:04.458 INFO:teuthology.orchestra.run.smithi084.stdout: protobuf-3.14.0-13.el9.x86_64 2024-03-29T11:18:04.458 INFO:teuthology.orchestra.run.smithi084.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-03-29T11:18:04.458 INFO:teuthology.orchestra.run.smithi084.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-03-29T11:18:04.458 INFO:teuthology.orchestra.run.smithi084.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-03-29T11:18:04.458 INFO:teuthology.orchestra.run.smithi084.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-03-29T11:18:04.458 INFO:teuthology.orchestra.run.smithi084.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-29T11:18:04.459 INFO:teuthology.orchestra.run.smithi084.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-29T11:18:04.459 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-03-29T11:18:04.459 INFO:teuthology.orchestra.run.smithi084.stdout:Complete! 2024-03-29T11:18:04.476 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-03-29T11:18:04.583 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-03-29T11:18:04.714 DEBUG:teuthology.orchestra.run.smithi084:> sudo yum -y install ceph-mgr-diskprediction-local 2024-03-29T11:18:05.277 INFO:teuthology.orchestra.run.smithi084.stdout:Last metadata expiration check: 0:01:53 ago on Fri 29 Mar 2024 11:16:12 AM UTC. 2024-03-29T11:18:05.429 INFO:teuthology.orchestra.run.smithi084.stdout:Dependencies resolved. 2024-03-29T11:18:05.430 INFO:teuthology.orchestra.run.smithi084.stdout:============================================================================================ 2024-03-29T11:18:05.430 INFO:teuthology.orchestra.run.smithi084.stdout: Package Arch Version Repository Size 2024-03-29T11:18:05.430 INFO:teuthology.orchestra.run.smithi084.stdout:============================================================================================ 2024-03-29T11:18:05.430 INFO:teuthology.orchestra.run.smithi084.stdout:Installing: 2024-03-29T11:18:05.430 INFO:teuthology.orchestra.run.smithi084.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 7.4 M 2024-03-29T11:18:05.430 INFO:teuthology.orchestra.run.smithi084.stdout:Installing dependencies: 2024-03-29T11:18:05.430 INFO:teuthology.orchestra.run.smithi084.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-29T11:18:05.431 INFO:teuthology.orchestra.run.smithi084.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-29T11:18:05.431 INFO:teuthology.orchestra.run.smithi084.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-29T11:18:05.431 INFO:teuthology.orchestra.run.smithi084.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-03-29T11:18:05.431 INFO:teuthology.orchestra.run.smithi084.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-03-29T11:18:05.431 INFO:teuthology.orchestra.run.smithi084.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-29T11:18:05.431 INFO:teuthology.orchestra.run.smithi084.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-29T11:18:05.431 INFO:teuthology.orchestra.run.smithi084.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-29T11:18:05.431 INFO:teuthology.orchestra.run.smithi084.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-29T11:18:05.431 INFO:teuthology.orchestra.run.smithi084.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-29T11:18:05.431 INFO:teuthology.orchestra.run.smithi084.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-29T11:18:05.431 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-03-29T11:18:05.431 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction Summary 2024-03-29T11:18:05.431 INFO:teuthology.orchestra.run.smithi084.stdout:============================================================================================ 2024-03-29T11:18:05.432 INFO:teuthology.orchestra.run.smithi084.stdout:Install 12 Packages 2024-03-29T11:18:05.432 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-03-29T11:18:05.432 INFO:teuthology.orchestra.run.smithi084.stdout:Total download size: 37 M 2024-03-29T11:18:05.432 INFO:teuthology.orchestra.run.smithi084.stdout:Installed size: 209 M 2024-03-29T11:18:05.432 INFO:teuthology.orchestra.run.smithi084.stdout:Downloading Packages: 2024-03-29T11:18:05.741 INFO:teuthology.orchestra.run.smithi084.stdout:(1/12): flexiblas-3.0.4-8.el9.x86_64.rpm 223 kB/s | 32 kB 00:00 2024-03-29T11:18:05.775 INFO:teuthology.orchestra.run.smithi084.stdout:(2/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 521 kB/s | 17 kB 00:00 2024-03-29T11:18:05.808 INFO:teuthology.orchestra.run.smithi084.stdout:(3/12): openblas-0.3.21-2.el9.x86_64.rpm 1.0 MB/s | 34 kB 00:00 2024-03-29T11:18:05.862 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-03-29T11:18:06.142 INFO:teuthology.orchestra.run.smithi084.stdout:(4/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 5.5 MB/s | 3.0 MB 00:00 2024-03-29T11:18:06.734 INFO:teuthology.orchestra.run.smithi084.stdout:(5/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 5.1 MB/s | 4.7 MB 00:00 2024-03-29T11:18:06.876 INFO:teuthology.orchestra.run.smithi084.stdout:(6/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 6.5 MB/s | 4.8 MB 00:00 2024-03-29T11:18:06.893 INFO:teuthology.orchestra.run.smithi084.stdout:(7/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 1.4 MB/s | 217 kB 00:00 2024-03-29T11:18:06.896 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-03-29T11:18:06.928 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 12/12 2024-03-29T11:18:07.085 INFO:teuthology.orchestra.run.smithi084.stdout:(8/12): libgfortran-11.4.1-3.el9.x86_64.rpm 4.1 MB/s | 803 kB 00:00 2024-03-29T11:18:07.127 INFO:teuthology.orchestra.run.smithi084.stdout:(9/12): libquadmath-11.4.1-3.el9.x86_64.rpm 4.9 MB/s | 195 kB 00:00 2024-03-29T11:18:07.327 INFO:teuthology.orchestra.run.smithi084.stdout:(10/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 35 MB/s | 16 MB 00:00 2024-03-29T11:18:07.377 INFO:teuthology.orchestra.run.smithi084.stdout:(11/12): python3-devel-3.9.17-2.el9.x86_64.rpm 990 kB/s | 245 kB 00:00 2024-03-29T11:18:08.210 INFO:teuthology.orchestra.run.smithi084.stdout:(12/12): ceph-mgr-diskprediction-local-19.0.0-1 2.8 MB/s | 7.4 MB 00:02 2024-03-29T11:18:08.213 INFO:teuthology.orchestra.run.smithi084.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:18:08.213 INFO:teuthology.orchestra.run.smithi084.stdout:Total 13 MB/s | 37 MB 00:02 2024-03-29T11:18:08.255 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction check 2024-03-29T11:18:08.278 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction check succeeded. 2024-03-29T11:18:08.279 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction test 2024-03-29T11:18:08.638 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction test succeeded. 2024-03-29T11:18:08.638 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction 2024-03-29T11:18:09.352 INFO:teuthology.orchestra.run.smithi084.stdout: Preparing : 1/1 2024-03-29T11:18:09.417 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 1/12 2024-03-29T11:18:09.456 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-03-29T11:18:09.541 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 3/12 2024-03-29T11:18:09.581 INFO:teuthology.orchestra.run.smithi179.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 12/12 2024-03-29T11:18:09.581 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 1/12 2024-03-29T11:18:09.581 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 2/12 2024-03-29T11:18:09.581 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 3/12 2024-03-29T11:18:09.581 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 4/12 2024-03-29T11:18:09.581 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-29T11:18:09.581 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-29T11:18:09.581 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 7/12 2024-03-29T11:18:09.581 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 8/12 2024-03-29T11:18:09.581 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 9/12 2024-03-29T11:18:09.582 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 10/12 2024-03-29T11:18:09.582 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 11/12 2024-03-29T11:18:09.592 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 4/12 2024-03-29T11:18:09.762 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-29T11:18:09.792 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-29T11:18:09.881 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-03-29T11:18:10.067 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 12/12 2024-03-29T11:18:10.068 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-29T11:18:10.068 INFO:teuthology.orchestra.run.smithi179.stdout:Installed: 2024-03-29T11:18:10.068 INFO:teuthology.orchestra.run.smithi179.stdout: ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:18:10.068 INFO:teuthology.orchestra.run.smithi179.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-29T11:18:10.068 INFO:teuthology.orchestra.run.smithi179.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-29T11:18:10.068 INFO:teuthology.orchestra.run.smithi179.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-29T11:18:10.068 INFO:teuthology.orchestra.run.smithi179.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-29T11:18:10.068 INFO:teuthology.orchestra.run.smithi179.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-29T11:18:10.068 INFO:teuthology.orchestra.run.smithi179.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-29T11:18:10.068 INFO:teuthology.orchestra.run.smithi179.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-29T11:18:10.068 INFO:teuthology.orchestra.run.smithi179.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-29T11:18:10.068 INFO:teuthology.orchestra.run.smithi179.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-29T11:18:10.069 INFO:teuthology.orchestra.run.smithi179.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-29T11:18:10.069 INFO:teuthology.orchestra.run.smithi179.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-29T11:18:10.069 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-29T11:18:10.069 INFO:teuthology.orchestra.run.smithi179.stdout:Complete! 2024-03-29T11:18:10.416 DEBUG:teuthology.orchestra.run.smithi179:> sudo yum -y install ceph-mgr-rook 2024-03-29T11:18:10.535 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-03-29T11:18:10.652 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-03-29T11:18:10.986 INFO:teuthology.orchestra.run.smithi179.stdout:Last metadata expiration check: 0:01:54 ago on Fri 29 Mar 2024 11:16:17 AM UTC. 2024-03-29T11:18:11.137 INFO:teuthology.orchestra.run.smithi179.stdout:Dependencies resolved. 2024-03-29T11:18:11.139 INFO:teuthology.orchestra.run.smithi179.stdout:======================================================================================== 2024-03-29T11:18:11.139 INFO:teuthology.orchestra.run.smithi179.stdout: Package Arch Version Repository Size 2024-03-29T11:18:11.139 INFO:teuthology.orchestra.run.smithi179.stdout:======================================================================================== 2024-03-29T11:18:11.139 INFO:teuthology.orchestra.run.smithi179.stdout:Installing: 2024-03-29T11:18:11.139 INFO:teuthology.orchestra.run.smithi179.stdout: ceph-mgr-rook noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 52 k 2024-03-29T11:18:11.139 INFO:teuthology.orchestra.run.smithi179.stdout:Installing dependencies: 2024-03-29T11:18:11.139 INFO:teuthology.orchestra.run.smithi179.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-29T11:18:11.139 INFO:teuthology.orchestra.run.smithi179.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-29T11:18:11.139 INFO:teuthology.orchestra.run.smithi179.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-03-29T11:18:11.139 INFO:teuthology.orchestra.run.smithi179.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-29T11:18:11.139 INFO:teuthology.orchestra.run.smithi179.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-29T11:18:11.139 INFO:teuthology.orchestra.run.smithi179.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-29T11:18:11.140 INFO:teuthology.orchestra.run.smithi179.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-29T11:18:11.140 INFO:teuthology.orchestra.run.smithi179.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-29T11:18:11.140 INFO:teuthology.orchestra.run.smithi179.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-29T11:18:11.140 INFO:teuthology.orchestra.run.smithi179.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-29T11:18:11.140 INFO:teuthology.orchestra.run.smithi179.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-29T11:18:11.140 INFO:teuthology.orchestra.run.smithi179.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-29T11:18:11.140 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-29T11:18:11.140 INFO:teuthology.orchestra.run.smithi179.stdout:Transaction Summary 2024-03-29T11:18:11.140 INFO:teuthology.orchestra.run.smithi179.stdout:======================================================================================== 2024-03-29T11:18:11.140 INFO:teuthology.orchestra.run.smithi179.stdout:Install 13 Packages 2024-03-29T11:18:11.140 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-29T11:18:11.141 INFO:teuthology.orchestra.run.smithi179.stdout:Total download size: 2.0 M 2024-03-29T11:18:11.141 INFO:teuthology.orchestra.run.smithi179.stdout:Installed size: 26 M 2024-03-29T11:18:11.141 INFO:teuthology.orchestra.run.smithi179.stdout:Downloading Packages: 2024-03-29T11:18:11.403 INFO:teuthology.orchestra.run.smithi179.stdout:(1/13): python3-jsonpatch-1.21-16.el9.noarch.rp 235 kB/s | 24 kB 00:00 2024-03-29T11:18:11.453 INFO:teuthology.orchestra.run.smithi179.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 117 kB/s | 18 kB 00:00 2024-03-29T11:18:11.495 INFO:teuthology.orchestra.run.smithi179.stdout:(3/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 2.5 MB/s | 174 kB 00:00 2024-03-29T11:18:11.520 INFO:teuthology.orchestra.run.smithi179.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.9 MB/s | 132 kB 00:00 2024-03-29T11:18:11.545 INFO:teuthology.orchestra.run.smithi179.stdout:(5/13): python3-pyasn1-modules-0.4.8-6.el9.noar 4.1 MB/s | 211 kB 00:00 2024-03-29T11:18:11.570 INFO:teuthology.orchestra.run.smithi179.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 871 kB/s | 43 kB 00:00 2024-03-29T11:18:11.704 INFO:teuthology.orchestra.run.smithi179.stdout:(7/13): python3-cachetools-4.2.4-1.el9.noarch.r 203 kB/s | 32 kB 00:00 2024-03-29T11:18:11.729 INFO:teuthology.orchestra.run.smithi179.stdout:(8/13): python3-certifi-2021.10.8-2.el9.noarch. 94 kB/s | 15 kB 00:00 2024-03-29T11:18:11.788 INFO:teuthology.orchestra.run.smithi179.stdout:(9/13): python3-google-auth-2.28.2-1.el9.noarch 2.4 MB/s | 203 kB 00:00 2024-03-29T11:18:11.838 INFO:teuthology.orchestra.run.smithi179.stdout:(10/13): python3-kubernetes-26.1.0-2.el9.noarch 9.4 MB/s | 1.0 MB 00:00 2024-03-29T11:18:11.864 INFO:teuthology.orchestra.run.smithi179.stdout:(11/13): python3-rsa-4.9-2.el9.noarch.rpm 790 kB/s | 59 kB 00:00 2024-03-29T11:18:11.884 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-03-29T11:18:11.889 INFO:teuthology.orchestra.run.smithi179.stdout:(12/13): ceph-mgr-rook-19.0.0-1439.g5dcb190d.el 88 kB/s | 52 kB 00:00 2024-03-29T11:18:11.914 INFO:teuthology.orchestra.run.smithi179.stdout:(13/13): python3-websocket-client-1.2.3-2.el9.n 1.2 MB/s | 90 kB 00:00 2024-03-29T11:18:11.917 INFO:teuthology.orchestra.run.smithi179.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:18:11.918 INFO:teuthology.orchestra.run.smithi179.stdout:Total 2.6 MB/s | 2.0 MB 00:00 2024-03-29T11:18:11.918 INFO:teuthology.orchestra.run.smithi179.stdout:Running transaction check 2024-03-29T11:18:11.931 INFO:teuthology.orchestra.run.smithi179.stdout:Transaction check succeeded. 2024-03-29T11:18:11.931 INFO:teuthology.orchestra.run.smithi179.stdout:Running transaction test 2024-03-29T11:18:12.021 INFO:teuthology.orchestra.run.smithi179.stdout:Transaction test succeeded. 2024-03-29T11:18:12.021 INFO:teuthology.orchestra.run.smithi179.stdout:Running transaction 2024-03-29T11:18:12.542 INFO:teuthology.orchestra.run.smithi179.stdout: Preparing : 1/1 2024-03-29T11:18:12.713 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-03-29T11:18:12.773 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-03-29T11:18:12.838 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-03-29T11:18:12.874 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-03-29T11:18:12.909 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-03-29T11:18:12.915 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2024-03-29T11:18:12.940 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 12/12 2024-03-29T11:18:13.009 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-03-29T11:18:13.120 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 7/13 2024-03-29T11:18:13.169 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-03-29T11:18:13.896 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-03-29T11:18:13.949 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-03-29T11:18:13.983 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-03-29T11:18:14.031 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-03-29T11:18:14.053 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 13/13 2024-03-29T11:18:14.907 INFO:teuthology.orchestra.run.smithi179.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 13/13 2024-03-29T11:18:14.907 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 1/13 2024-03-29T11:18:14.907 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-03-29T11:18:14.907 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-03-29T11:18:14.907 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-03-29T11:18:14.907 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-03-29T11:18:14.907 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-03-29T11:18:14.907 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-03-29T11:18:14.907 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-03-29T11:18:14.907 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2024-03-29T11:18:14.907 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 10/13 2024-03-29T11:18:14.908 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-03-29T11:18:14.908 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-03-29T11:18:15.264 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-03-29T11:18:15.265 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-29T11:18:15.265 INFO:teuthology.orchestra.run.smithi179.stdout:Installed: 2024-03-29T11:18:15.265 INFO:teuthology.orchestra.run.smithi179.stdout: ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:18:15.265 INFO:teuthology.orchestra.run.smithi179.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-29T11:18:15.265 INFO:teuthology.orchestra.run.smithi179.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-29T11:18:15.265 INFO:teuthology.orchestra.run.smithi179.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-03-29T11:18:15.265 INFO:teuthology.orchestra.run.smithi179.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-29T11:18:15.265 INFO:teuthology.orchestra.run.smithi179.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-29T11:18:15.265 INFO:teuthology.orchestra.run.smithi179.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-29T11:18:15.265 INFO:teuthology.orchestra.run.smithi179.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-29T11:18:15.265 INFO:teuthology.orchestra.run.smithi179.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-29T11:18:15.265 INFO:teuthology.orchestra.run.smithi179.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-29T11:18:15.265 INFO:teuthology.orchestra.run.smithi179.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-29T11:18:15.265 INFO:teuthology.orchestra.run.smithi179.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-29T11:18:15.266 INFO:teuthology.orchestra.run.smithi179.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-29T11:18:15.266 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-29T11:18:15.266 INFO:teuthology.orchestra.run.smithi179.stdout:Complete! 2024-03-29T11:18:15.367 INFO:teuthology.orchestra.run.smithi084.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 12/12 2024-03-29T11:18:15.367 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 1/12 2024-03-29T11:18:15.367 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 2/12 2024-03-29T11:18:15.367 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 3/12 2024-03-29T11:18:15.367 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 4/12 2024-03-29T11:18:15.367 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-29T11:18:15.367 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-29T11:18:15.367 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 7/12 2024-03-29T11:18:15.367 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 8/12 2024-03-29T11:18:15.368 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 9/12 2024-03-29T11:18:15.368 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 10/12 2024-03-29T11:18:15.368 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 11/12 2024-03-29T11:18:15.497 DEBUG:teuthology.orchestra.run.smithi179:> sudo yum -y install ceph-mgr-cephadm 2024-03-29T11:18:15.803 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 12/12 2024-03-29T11:18:15.803 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-03-29T11:18:15.803 INFO:teuthology.orchestra.run.smithi084.stdout:Installed: 2024-03-29T11:18:15.803 INFO:teuthology.orchestra.run.smithi084.stdout: ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:18:15.804 INFO:teuthology.orchestra.run.smithi084.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-29T11:18:15.804 INFO:teuthology.orchestra.run.smithi084.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-29T11:18:15.804 INFO:teuthology.orchestra.run.smithi084.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-29T11:18:15.804 INFO:teuthology.orchestra.run.smithi084.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-29T11:18:15.804 INFO:teuthology.orchestra.run.smithi084.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-29T11:18:15.804 INFO:teuthology.orchestra.run.smithi084.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-29T11:18:15.804 INFO:teuthology.orchestra.run.smithi084.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-29T11:18:15.804 INFO:teuthology.orchestra.run.smithi084.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-29T11:18:15.804 INFO:teuthology.orchestra.run.smithi084.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-29T11:18:15.804 INFO:teuthology.orchestra.run.smithi084.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-29T11:18:15.804 INFO:teuthology.orchestra.run.smithi084.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-29T11:18:15.804 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-03-29T11:18:15.804 INFO:teuthology.orchestra.run.smithi084.stdout:Complete! 2024-03-29T11:18:16.068 INFO:teuthology.orchestra.run.smithi179.stdout:Last metadata expiration check: 0:01:59 ago on Fri 29 Mar 2024 11:16:17 AM UTC. 2024-03-29T11:18:16.204 DEBUG:teuthology.orchestra.run.smithi084:> sudo yum -y install ceph-mgr-rook 2024-03-29T11:18:16.220 INFO:teuthology.orchestra.run.smithi179.stdout:Dependencies resolved. 2024-03-29T11:18:16.221 INFO:teuthology.orchestra.run.smithi179.stdout:======================================================================================== 2024-03-29T11:18:16.221 INFO:teuthology.orchestra.run.smithi179.stdout: Package Arch Version Repository Size 2024-03-29T11:18:16.221 INFO:teuthology.orchestra.run.smithi179.stdout:======================================================================================== 2024-03-29T11:18:16.221 INFO:teuthology.orchestra.run.smithi179.stdout:Installing: 2024-03-29T11:18:16.222 INFO:teuthology.orchestra.run.smithi179.stdout: ceph-mgr-cephadm noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 138 k 2024-03-29T11:18:16.222 INFO:teuthology.orchestra.run.smithi179.stdout:Installing dependencies: 2024-03-29T11:18:16.222 INFO:teuthology.orchestra.run.smithi179.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-03-29T11:18:16.222 INFO:teuthology.orchestra.run.smithi179.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-29T11:18:16.222 INFO:teuthology.orchestra.run.smithi179.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-03-29T11:18:16.222 INFO:teuthology.orchestra.run.smithi179.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-29T11:18:16.222 INFO:teuthology.orchestra.run.smithi179.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-29T11:18:16.222 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-29T11:18:16.223 INFO:teuthology.orchestra.run.smithi179.stdout:Transaction Summary 2024-03-29T11:18:16.223 INFO:teuthology.orchestra.run.smithi179.stdout:======================================================================================== 2024-03-29T11:18:16.223 INFO:teuthology.orchestra.run.smithi179.stdout:Install 6 Packages 2024-03-29T11:18:16.223 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-29T11:18:16.224 INFO:teuthology.orchestra.run.smithi179.stdout:Total download size: 6.8 M 2024-03-29T11:18:16.224 INFO:teuthology.orchestra.run.smithi179.stdout:Installed size: 33 M 2024-03-29T11:18:16.224 INFO:teuthology.orchestra.run.smithi179.stdout:Downloading Packages: 2024-03-29T11:18:16.559 INFO:teuthology.orchestra.run.smithi179.stdout:(1/6): python3-jinja2-2.11.3-5.el9.noarch.rpm 2.6 MB/s | 248 kB 00:00 2024-03-29T11:18:16.743 INFO:teuthology.orchestra.run.smithi179.stdout:(2/6): python3-babel-2.9.1-2.el9.noarch.rpm 21 MB/s | 5.8 MB 00:00 2024-03-29T11:18:16.772 INFO:teuthology.orchestra.run.smithi084.stdout:Last metadata expiration check: 0:02:04 ago on Fri 29 Mar 2024 11:16:12 AM UTC. 2024-03-29T11:18:16.923 INFO:teuthology.orchestra.run.smithi084.stdout:Dependencies resolved. 2024-03-29T11:18:16.924 INFO:teuthology.orchestra.run.smithi084.stdout:======================================================================================== 2024-03-29T11:18:16.924 INFO:teuthology.orchestra.run.smithi084.stdout: Package Arch Version Repository Size 2024-03-29T11:18:16.924 INFO:teuthology.orchestra.run.smithi084.stdout:======================================================================================== 2024-03-29T11:18:16.924 INFO:teuthology.orchestra.run.smithi084.stdout:Installing: 2024-03-29T11:18:16.924 INFO:teuthology.orchestra.run.smithi084.stdout: ceph-mgr-rook noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 52 k 2024-03-29T11:18:16.924 INFO:teuthology.orchestra.run.smithi084.stdout:Installing dependencies: 2024-03-29T11:18:16.924 INFO:teuthology.orchestra.run.smithi084.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-29T11:18:16.924 INFO:teuthology.orchestra.run.smithi084.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-29T11:18:16.924 INFO:teuthology.orchestra.run.smithi084.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-03-29T11:18:16.924 INFO:teuthology.orchestra.run.smithi084.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-29T11:18:16.925 INFO:teuthology.orchestra.run.smithi084.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-29T11:18:16.925 INFO:teuthology.orchestra.run.smithi084.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-29T11:18:16.925 INFO:teuthology.orchestra.run.smithi084.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-29T11:18:16.925 INFO:teuthology.orchestra.run.smithi084.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-29T11:18:16.925 INFO:teuthology.orchestra.run.smithi084.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-29T11:18:16.925 INFO:teuthology.orchestra.run.smithi084.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-29T11:18:16.925 INFO:teuthology.orchestra.run.smithi084.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-29T11:18:16.925 INFO:teuthology.orchestra.run.smithi084.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-29T11:18:16.925 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-03-29T11:18:16.925 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction Summary 2024-03-29T11:18:16.925 INFO:teuthology.orchestra.run.smithi084.stdout:======================================================================================== 2024-03-29T11:18:16.925 INFO:teuthology.orchestra.run.smithi084.stdout:Install 13 Packages 2024-03-29T11:18:16.925 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-03-29T11:18:16.926 INFO:teuthology.orchestra.run.smithi084.stdout:Total download size: 2.0 M 2024-03-29T11:18:16.926 INFO:teuthology.orchestra.run.smithi084.stdout:Installed size: 26 M 2024-03-29T11:18:16.926 INFO:teuthology.orchestra.run.smithi084.stdout:Downloading Packages: 2024-03-29T11:18:17.068 INFO:teuthology.orchestra.run.smithi179.stdout:(3/6): python3-natsort-7.1.1-5.el9.noarch.rpm 178 kB/s | 58 kB 00:00 2024-03-29T11:18:17.131 INFO:teuthology.orchestra.run.smithi084.stdout:(1/13): python3-jsonpointer-2.0-4.el9.noarch.rp 140 kB/s | 18 kB 00:00 2024-03-29T11:18:17.156 INFO:teuthology.orchestra.run.smithi084.stdout:(2/13): python3-jsonpatch-1.21-16.el9.noarch.rp 158 kB/s | 24 kB 00:00 2024-03-29T11:18:17.202 INFO:teuthology.orchestra.run.smithi179.stdout:(4/6): python3-typing-extensions-4.4.0-2.el9.no 383 kB/s | 51 kB 00:00 2024-03-29T11:18:17.206 INFO:teuthology.orchestra.run.smithi084.stdout:(3/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 2.3 MB/s | 174 kB 00:00 2024-03-29T11:18:17.227 INFO:teuthology.orchestra.run.smithi179.stdout:(5/6): ceph-mgr-cephadm-19.0.0-1439.g5dcb190d.e 181 kB/s | 138 kB 00:00 2024-03-29T11:18:17.231 INFO:teuthology.orchestra.run.smithi084.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.7 MB/s | 132 kB 00:00 2024-03-29T11:18:17.252 INFO:teuthology.orchestra.run.smithi179.stdout:(6/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 819 kB/s | 548 kB 00:00 2024-03-29T11:18:17.253 INFO:teuthology.orchestra.run.smithi179.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:18:17.254 INFO:teuthology.orchestra.run.smithi179.stdout:Total 6.6 MB/s | 6.8 MB 00:01 2024-03-29T11:18:17.256 INFO:teuthology.orchestra.run.smithi084.stdout:(5/13): python3-pyasn1-modules-0.4.8-6.el9.noar 4.1 MB/s | 211 kB 00:00 2024-03-29T11:18:17.269 INFO:teuthology.orchestra.run.smithi179.stdout:Running transaction check 2024-03-29T11:18:17.278 INFO:teuthology.orchestra.run.smithi179.stdout:Transaction check succeeded. 2024-03-29T11:18:17.278 INFO:teuthology.orchestra.run.smithi179.stdout:Running transaction test 2024-03-29T11:18:17.282 INFO:teuthology.orchestra.run.smithi084.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 859 kB/s | 43 kB 00:00 2024-03-29T11:18:17.366 INFO:teuthology.orchestra.run.smithi179.stdout:Transaction test succeeded. 2024-03-29T11:18:17.366 INFO:teuthology.orchestra.run.smithi179.stdout:Running transaction 2024-03-29T11:18:17.407 INFO:teuthology.orchestra.run.smithi084.stdout:(7/13): ceph-mgr-rook-19.0.0-1439.g5dcb190d.el9 129 kB/s | 52 kB 00:00 2024-03-29T11:18:17.524 INFO:teuthology.orchestra.run.smithi084.stdout:(8/13): python3-certifi-2021.10.8-2.el9.noarch. 62 kB/s | 15 kB 00:00 2024-03-29T11:18:17.549 INFO:teuthology.orchestra.run.smithi084.stdout:(9/13): python3-cachetools-4.2.4-1.el9.noarch.r 110 kB/s | 32 kB 00:00 2024-03-29T11:18:17.660 INFO:teuthology.orchestra.run.smithi179.stdout: Preparing : 1/1 2024-03-29T11:18:17.758 INFO:teuthology.orchestra.run.smithi084.stdout:(10/13): python3-rsa-4.9-2.el9.noarch.rpm 284 kB/s | 59 kB 00:00 2024-03-29T11:18:17.835 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-03-29T11:18:17.887 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-03-29T11:18:17.900 INFO:teuthology.orchestra.run.smithi084.stdout:(11/13): python3-google-auth-2.28.2-1.el9.noarc 411 kB/s | 203 kB 00:00 2024-03-29T11:18:17.925 INFO:teuthology.orchestra.run.smithi084.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 538 kB/s | 90 kB 00:00 2024-03-29T11:18:18.067 INFO:teuthology.orchestra.run.smithi084.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 1.9 MB/s | 1.0 MB 00:00 2024-03-29T11:18:18.069 INFO:teuthology.orchestra.run.smithi084.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:18:18.070 INFO:teuthology.orchestra.run.smithi084.stdout:Total 1.8 MB/s | 2.0 MB 00:01 2024-03-29T11:18:18.070 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction check 2024-03-29T11:18:18.085 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction check succeeded. 2024-03-29T11:18:18.085 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction test 2024-03-29T11:18:18.183 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction test succeeded. 2024-03-29T11:18:18.183 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction 2024-03-29T11:18:18.398 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-03-29T11:18:18.472 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-03-29T11:18:18.534 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 5/6 2024-03-29T11:18:18.557 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 6/6 2024-03-29T11:18:18.721 INFO:teuthology.orchestra.run.smithi084.stdout: Preparing : 1/1 2024-03-29T11:18:18.880 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-03-29T11:18:18.942 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-03-29T11:18:19.006 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-03-29T11:18:19.069 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-03-29T11:18:19.119 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2024-03-29T11:18:19.212 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-03-29T11:18:19.313 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 7/13 2024-03-29T11:18:19.354 INFO:teuthology.orchestra.run.smithi179.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 6/6 2024-03-29T11:18:19.354 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/6 2024-03-29T11:18:19.354 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-03-29T11:18:19.354 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 3/6 2024-03-29T11:18:19.354 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-03-29T11:18:19.354 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-03-29T11:18:19.370 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-03-29T11:18:19.655 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-03-29T11:18:19.655 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-29T11:18:19.655 INFO:teuthology.orchestra.run.smithi179.stdout:Installed: 2024-03-29T11:18:19.655 INFO:teuthology.orchestra.run.smithi179.stdout: ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:18:19.655 INFO:teuthology.orchestra.run.smithi179.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-03-29T11:18:19.655 INFO:teuthology.orchestra.run.smithi179.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-29T11:18:19.655 INFO:teuthology.orchestra.run.smithi179.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-29T11:18:19.656 INFO:teuthology.orchestra.run.smithi179.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-29T11:18:19.656 INFO:teuthology.orchestra.run.smithi179.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-29T11:18:19.656 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-29T11:18:19.656 INFO:teuthology.orchestra.run.smithi179.stdout:Complete! 2024-03-29T11:18:19.895 DEBUG:teuthology.orchestra.run.smithi179:> sudo yum -y install ceph-fuse 2024-03-29T11:18:20.090 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-03-29T11:18:20.134 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-03-29T11:18:20.167 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-03-29T11:18:20.216 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-03-29T11:18:20.238 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 13/13 2024-03-29T11:18:20.456 INFO:teuthology.orchestra.run.smithi179.stdout:Last metadata expiration check: 0:02:03 ago on Fri 29 Mar 2024 11:16:17 AM UTC. 2024-03-29T11:18:20.606 INFO:teuthology.orchestra.run.smithi179.stdout:Dependencies resolved. 2024-03-29T11:18:20.606 INFO:teuthology.orchestra.run.smithi179.stdout:================================================================================ 2024-03-29T11:18:20.606 INFO:teuthology.orchestra.run.smithi179.stdout: Package Arch Version Repository Size 2024-03-29T11:18:20.606 INFO:teuthology.orchestra.run.smithi179.stdout:================================================================================ 2024-03-29T11:18:20.607 INFO:teuthology.orchestra.run.smithi179.stdout:Installing: 2024-03-29T11:18:20.607 INFO:teuthology.orchestra.run.smithi179.stdout: ceph-fuse x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 885 k 2024-03-29T11:18:20.607 INFO:teuthology.orchestra.run.smithi179.stdout:Installing dependencies: 2024-03-29T11:18:20.607 INFO:teuthology.orchestra.run.smithi179.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-29T11:18:20.607 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-29T11:18:20.607 INFO:teuthology.orchestra.run.smithi179.stdout:Transaction Summary 2024-03-29T11:18:20.607 INFO:teuthology.orchestra.run.smithi179.stdout:================================================================================ 2024-03-29T11:18:20.607 INFO:teuthology.orchestra.run.smithi179.stdout:Install 2 Packages 2024-03-29T11:18:20.607 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-29T11:18:20.607 INFO:teuthology.orchestra.run.smithi179.stdout:Total download size: 964 k 2024-03-29T11:18:20.608 INFO:teuthology.orchestra.run.smithi179.stdout:Installed size: 2.8 M 2024-03-29T11:18:20.608 INFO:teuthology.orchestra.run.smithi179.stdout:Downloading Packages: 2024-03-29T11:18:20.728 INFO:teuthology.orchestra.run.smithi179.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 654 kB/s | 79 kB 00:00 2024-03-29T11:18:20.953 INFO:teuthology.orchestra.run.smithi179.stdout:(2/2): ceph-fuse-19.0.0-1439.g5dcb190d.el9.x86_ 2.5 MB/s | 885 kB 00:00 2024-03-29T11:18:20.955 INFO:teuthology.orchestra.run.smithi179.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:18:20.955 INFO:teuthology.orchestra.run.smithi179.stdout:Total 2.7 MB/s | 964 kB 00:00 2024-03-29T11:18:20.955 INFO:teuthology.orchestra.run.smithi179.stdout:Running transaction check 2024-03-29T11:18:20.975 INFO:teuthology.orchestra.run.smithi179.stdout:Transaction check succeeded. 2024-03-29T11:18:20.975 INFO:teuthology.orchestra.run.smithi179.stdout:Running transaction test 2024-03-29T11:18:21.048 INFO:teuthology.orchestra.run.smithi179.stdout:Transaction test succeeded. 2024-03-29T11:18:21.048 INFO:teuthology.orchestra.run.smithi179.stdout:Running transaction 2024-03-29T11:18:21.112 INFO:teuthology.orchestra.run.smithi084.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 13/13 2024-03-29T11:18:21.112 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 1/13 2024-03-29T11:18:21.112 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-03-29T11:18:21.112 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-03-29T11:18:21.112 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-03-29T11:18:21.112 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-03-29T11:18:21.112 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-03-29T11:18:21.112 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-03-29T11:18:21.112 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-03-29T11:18:21.112 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2024-03-29T11:18:21.112 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 10/13 2024-03-29T11:18:21.112 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-03-29T11:18:21.112 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-03-29T11:18:21.226 INFO:teuthology.orchestra.run.smithi179.stdout: Preparing : 1/1 2024-03-29T11:18:21.372 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-03-29T11:18:21.445 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/2 2024-03-29T11:18:21.465 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-03-29T11:18:21.465 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-03-29T11:18:21.465 INFO:teuthology.orchestra.run.smithi084.stdout:Installed: 2024-03-29T11:18:21.465 INFO:teuthology.orchestra.run.smithi084.stdout: ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:18:21.465 INFO:teuthology.orchestra.run.smithi084.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-29T11:18:21.465 INFO:teuthology.orchestra.run.smithi084.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-29T11:18:21.465 INFO:teuthology.orchestra.run.smithi084.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-03-29T11:18:21.465 INFO:teuthology.orchestra.run.smithi084.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-29T11:18:21.465 INFO:teuthology.orchestra.run.smithi084.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-29T11:18:21.465 INFO:teuthology.orchestra.run.smithi084.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-29T11:18:21.465 INFO:teuthology.orchestra.run.smithi084.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-29T11:18:21.465 INFO:teuthology.orchestra.run.smithi084.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-29T11:18:21.465 INFO:teuthology.orchestra.run.smithi084.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-29T11:18:21.466 INFO:teuthology.orchestra.run.smithi084.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-29T11:18:21.466 INFO:teuthology.orchestra.run.smithi084.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-29T11:18:21.466 INFO:teuthology.orchestra.run.smithi084.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-29T11:18:21.466 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-03-29T11:18:21.466 INFO:teuthology.orchestra.run.smithi084.stdout:Complete! 2024-03-29T11:18:21.698 DEBUG:teuthology.orchestra.run.smithi084:> sudo yum -y install ceph-mgr-cephadm 2024-03-29T11:18:22.113 INFO:teuthology.orchestra.run.smithi179.stdout: Running scriptlet: ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/2 2024-03-29T11:18:22.113 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/2 2024-03-29T11:18:22.263 INFO:teuthology.orchestra.run.smithi084.stdout:Last metadata expiration check: 0:02:10 ago on Fri 29 Mar 2024 11:16:12 AM UTC. 2024-03-29T11:18:22.414 INFO:teuthology.orchestra.run.smithi084.stdout:Dependencies resolved. 2024-03-29T11:18:22.414 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-03-29T11:18:22.415 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-29T11:18:22.415 INFO:teuthology.orchestra.run.smithi179.stdout:Installed: 2024-03-29T11:18:22.415 INFO:teuthology.orchestra.run.smithi179.stdout: ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2024-03-29T11:18:22.415 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-29T11:18:22.415 INFO:teuthology.orchestra.run.smithi179.stdout:Complete! 2024-03-29T11:18:22.416 INFO:teuthology.orchestra.run.smithi084.stdout:======================================================================================== 2024-03-29T11:18:22.416 INFO:teuthology.orchestra.run.smithi084.stdout: Package Arch Version Repository Size 2024-03-29T11:18:22.416 INFO:teuthology.orchestra.run.smithi084.stdout:======================================================================================== 2024-03-29T11:18:22.416 INFO:teuthology.orchestra.run.smithi084.stdout:Installing: 2024-03-29T11:18:22.416 INFO:teuthology.orchestra.run.smithi084.stdout: ceph-mgr-cephadm noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 138 k 2024-03-29T11:18:22.416 INFO:teuthology.orchestra.run.smithi084.stdout:Installing dependencies: 2024-03-29T11:18:22.416 INFO:teuthology.orchestra.run.smithi084.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-03-29T11:18:22.416 INFO:teuthology.orchestra.run.smithi084.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-29T11:18:22.416 INFO:teuthology.orchestra.run.smithi084.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-03-29T11:18:22.416 INFO:teuthology.orchestra.run.smithi084.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-29T11:18:22.416 INFO:teuthology.orchestra.run.smithi084.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-29T11:18:22.416 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-03-29T11:18:22.416 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction Summary 2024-03-29T11:18:22.416 INFO:teuthology.orchestra.run.smithi084.stdout:======================================================================================== 2024-03-29T11:18:22.417 INFO:teuthology.orchestra.run.smithi084.stdout:Install 6 Packages 2024-03-29T11:18:22.417 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-03-29T11:18:22.417 INFO:teuthology.orchestra.run.smithi084.stdout:Total download size: 6.8 M 2024-03-29T11:18:22.417 INFO:teuthology.orchestra.run.smithi084.stdout:Installed size: 33 M 2024-03-29T11:18:22.417 INFO:teuthology.orchestra.run.smithi084.stdout:Downloading Packages: 2024-03-29T11:18:22.654 DEBUG:teuthology.orchestra.run.smithi179:> sudo yum -y install ceph-volume 2024-03-29T11:18:22.960 INFO:teuthology.orchestra.run.smithi084.stdout:(1/6): ceph-mgr-cephadm-19.0.0-1439.g5dcb190d.e 555 kB/s | 138 kB 00:00 2024-03-29T11:18:23.234 INFO:teuthology.orchestra.run.smithi179.stdout:Last metadata expiration check: 0:02:06 ago on Fri 29 Mar 2024 11:16:17 AM UTC. 2024-03-29T11:18:23.269 INFO:teuthology.orchestra.run.smithi084.stdout:(2/6): python3-babel-2.9.1-2.el9.noarch.rpm 10 MB/s | 5.8 MB 00:00 2024-03-29T11:18:23.386 INFO:teuthology.orchestra.run.smithi179.stdout:Dependencies resolved. 2024-03-29T11:18:23.387 INFO:teuthology.orchestra.run.smithi179.stdout:================================================================================ 2024-03-29T11:18:23.387 INFO:teuthology.orchestra.run.smithi179.stdout: Package Arch Version Repository Size 2024-03-29T11:18:23.387 INFO:teuthology.orchestra.run.smithi179.stdout:================================================================================ 2024-03-29T11:18:23.387 INFO:teuthology.orchestra.run.smithi179.stdout:Installing: 2024-03-29T11:18:23.387 INFO:teuthology.orchestra.run.smithi179.stdout: ceph-volume noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 264 k 2024-03-29T11:18:23.387 INFO:teuthology.orchestra.run.smithi179.stdout:Installing dependencies: 2024-03-29T11:18:23.387 INFO:teuthology.orchestra.run.smithi179.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-29T11:18:23.387 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-29T11:18:23.387 INFO:teuthology.orchestra.run.smithi179.stdout:Transaction Summary 2024-03-29T11:18:23.388 INFO:teuthology.orchestra.run.smithi179.stdout:================================================================================ 2024-03-29T11:18:23.388 INFO:teuthology.orchestra.run.smithi179.stdout:Install 2 Packages 2024-03-29T11:18:23.388 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-29T11:18:23.388 INFO:teuthology.orchestra.run.smithi179.stdout:Total download size: 332 k 2024-03-29T11:18:23.388 INFO:teuthology.orchestra.run.smithi179.stdout:Installed size: 1.5 M 2024-03-29T11:18:23.388 INFO:teuthology.orchestra.run.smithi179.stdout:Downloading Packages: 2024-03-29T11:18:23.516 INFO:teuthology.orchestra.run.smithi179.stdout:(1/2): python3-packaging-20.9-5.el9.noarch.rpm 538 kB/s | 69 kB 00:00 2024-03-29T11:18:23.610 INFO:teuthology.orchestra.run.smithi084.stdout:(3/6): python3-natsort-7.1.1-5.el9.noarch.rpm 169 kB/s | 58 kB 00:00 2024-03-29T11:18:23.652 INFO:teuthology.orchestra.run.smithi084.stdout:(4/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 821 kB/s | 548 kB 00:00 2024-03-29T11:18:23.677 INFO:teuthology.orchestra.run.smithi084.stdout:(5/6): python3-jinja2-2.11.3-5.el9.noarch.rpm 257 kB/s | 248 kB 00:00 2024-03-29T11:18:23.801 INFO:teuthology.orchestra.run.smithi179.stdout:(2/2): ceph-volume-19.0.0-1439.g5dcb190d.el9.no 640 kB/s | 264 kB 00:00 2024-03-29T11:18:23.803 INFO:teuthology.orchestra.run.smithi179.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:18:23.803 INFO:teuthology.orchestra.run.smithi179.stdout:Total 801 kB/s | 332 kB 00:00 2024-03-29T11:18:23.803 INFO:teuthology.orchestra.run.smithi179.stdout:Running transaction check 2024-03-29T11:18:23.811 INFO:teuthology.orchestra.run.smithi179.stdout:Transaction check succeeded. 2024-03-29T11:18:23.811 INFO:teuthology.orchestra.run.smithi179.stdout:Running transaction test 2024-03-29T11:18:23.849 INFO:teuthology.orchestra.run.smithi179.stdout:Transaction test succeeded. 2024-03-29T11:18:23.849 INFO:teuthology.orchestra.run.smithi179.stdout:Running transaction 2024-03-29T11:18:23.952 INFO:teuthology.orchestra.run.smithi084.stdout:(6/6): python3-typing-extensions-4.4.0-2.el9.no 149 kB/s | 51 kB 00:00 2024-03-29T11:18:23.954 INFO:teuthology.orchestra.run.smithi084.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:18:23.955 INFO:teuthology.orchestra.run.smithi084.stdout:Total 4.4 MB/s | 6.8 MB 00:01 2024-03-29T11:18:23.969 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction check 2024-03-29T11:18:23.980 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction check succeeded. 2024-03-29T11:18:23.980 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction test 2024-03-29T11:18:23.997 INFO:teuthology.orchestra.run.smithi179.stdout: Preparing : 1/1 2024-03-29T11:18:24.070 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction test succeeded. 2024-03-29T11:18:24.070 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction 2024-03-29T11:18:24.186 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-packaging-20.9-5.el9.noarch 1/2 2024-03-29T11:18:24.220 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2/2 2024-03-29T11:18:24.236 INFO:teuthology.orchestra.run.smithi179.stdout: Running scriptlet: ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2/2 2024-03-29T11:18:24.237 INFO:teuthology.orchestra.run.smithi179.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T11:18:24.237 INFO:teuthology.orchestra.run.smithi179.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-29T11:18:24.237 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-29T11:18:24.369 INFO:teuthology.orchestra.run.smithi084.stdout: Preparing : 1/1 2024-03-29T11:18:24.529 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-03-29T11:18:24.581 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-03-29T11:18:24.988 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 1/2 2024-03-29T11:18:25.092 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-03-29T11:18:25.175 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-03-29T11:18:25.236 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 5/6 2024-03-29T11:18:25.260 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 6/6 2024-03-29T11:18:25.338 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 2/2 2024-03-29T11:18:25.338 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-29T11:18:25.338 INFO:teuthology.orchestra.run.smithi179.stdout:Installed: 2024-03-29T11:18:25.338 INFO:teuthology.orchestra.run.smithi179.stdout: ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:18:25.338 INFO:teuthology.orchestra.run.smithi179.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-29T11:18:25.338 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-29T11:18:25.338 INFO:teuthology.orchestra.run.smithi179.stdout:Complete! 2024-03-29T11:18:25.684 DEBUG:teuthology.orchestra.run.smithi179:> sudo yum -y install librados-devel 2024-03-29T11:18:26.103 INFO:teuthology.orchestra.run.smithi084.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 6/6 2024-03-29T11:18:26.104 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/6 2024-03-29T11:18:26.104 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-03-29T11:18:26.104 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 3/6 2024-03-29T11:18:26.104 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-03-29T11:18:26.104 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-03-29T11:18:26.265 INFO:teuthology.orchestra.run.smithi179.stdout:Last metadata expiration check: 0:02:09 ago on Fri 29 Mar 2024 11:16:17 AM UTC. 2024-03-29T11:18:26.414 INFO:teuthology.orchestra.run.smithi179.stdout:Dependencies resolved. 2024-03-29T11:18:26.415 INFO:teuthology.orchestra.run.smithi179.stdout:================================================================================ 2024-03-29T11:18:26.415 INFO:teuthology.orchestra.run.smithi179.stdout: Package Arch Version Repo Size 2024-03-29T11:18:26.415 INFO:teuthology.orchestra.run.smithi179.stdout:================================================================================ 2024-03-29T11:18:26.415 INFO:teuthology.orchestra.run.smithi179.stdout:Installing: 2024-03-29T11:18:26.415 INFO:teuthology.orchestra.run.smithi179.stdout: librados-devel x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 127 k 2024-03-29T11:18:26.415 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-29T11:18:26.415 INFO:teuthology.orchestra.run.smithi179.stdout:Transaction Summary 2024-03-29T11:18:26.415 INFO:teuthology.orchestra.run.smithi179.stdout:================================================================================ 2024-03-29T11:18:26.415 INFO:teuthology.orchestra.run.smithi179.stdout:Install 1 Package 2024-03-29T11:18:26.416 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-29T11:18:26.416 INFO:teuthology.orchestra.run.smithi179.stdout:Total download size: 127 k 2024-03-29T11:18:26.416 INFO:teuthology.orchestra.run.smithi179.stdout:Installed size: 456 k 2024-03-29T11:18:26.416 INFO:teuthology.orchestra.run.smithi179.stdout:Downloading Packages: 2024-03-29T11:18:26.441 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-03-29T11:18:26.441 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-03-29T11:18:26.441 INFO:teuthology.orchestra.run.smithi084.stdout:Installed: 2024-03-29T11:18:26.442 INFO:teuthology.orchestra.run.smithi084.stdout: ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:18:26.442 INFO:teuthology.orchestra.run.smithi084.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-03-29T11:18:26.442 INFO:teuthology.orchestra.run.smithi084.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-29T11:18:26.442 INFO:teuthology.orchestra.run.smithi084.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-29T11:18:26.442 INFO:teuthology.orchestra.run.smithi084.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-29T11:18:26.442 INFO:teuthology.orchestra.run.smithi084.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-29T11:18:26.442 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-03-29T11:18:26.442 INFO:teuthology.orchestra.run.smithi084.stdout:Complete! 2024-03-29T11:18:26.667 DEBUG:teuthology.orchestra.run.smithi084:> sudo yum -y install ceph-fuse 2024-03-29T11:18:26.848 INFO:teuthology.orchestra.run.smithi179.stdout:librados-devel-19.0.0-1439.g5dcb190d.el9.x86_64 293 kB/s | 127 kB 00:00 2024-03-29T11:18:26.849 INFO:teuthology.orchestra.run.smithi179.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:18:26.849 INFO:teuthology.orchestra.run.smithi179.stdout:Total 292 kB/s | 127 kB 00:00 2024-03-29T11:18:26.849 INFO:teuthology.orchestra.run.smithi179.stdout:Running transaction check 2024-03-29T11:18:26.855 INFO:teuthology.orchestra.run.smithi179.stdout:Transaction check succeeded. 2024-03-29T11:18:26.855 INFO:teuthology.orchestra.run.smithi179.stdout:Running transaction test 2024-03-29T11:18:26.911 INFO:teuthology.orchestra.run.smithi179.stdout:Transaction test succeeded. 2024-03-29T11:18:26.911 INFO:teuthology.orchestra.run.smithi179.stdout:Running transaction 2024-03-29T11:18:27.065 INFO:teuthology.orchestra.run.smithi179.stdout: Preparing : 1/1 2024-03-29T11:18:27.180 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:18:27.230 INFO:teuthology.orchestra.run.smithi084.stdout:Last metadata expiration check: 0:02:15 ago on Fri 29 Mar 2024 11:16:12 AM UTC. 2024-03-29T11:18:27.378 INFO:teuthology.orchestra.run.smithi084.stdout:Dependencies resolved. 2024-03-29T11:18:27.379 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2024-03-29T11:18:27.379 INFO:teuthology.orchestra.run.smithi084.stdout: Package Arch Version Repository Size 2024-03-29T11:18:27.379 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2024-03-29T11:18:27.379 INFO:teuthology.orchestra.run.smithi084.stdout:Installing: 2024-03-29T11:18:27.379 INFO:teuthology.orchestra.run.smithi084.stdout: ceph-fuse x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 885 k 2024-03-29T11:18:27.379 INFO:teuthology.orchestra.run.smithi084.stdout:Installing dependencies: 2024-03-29T11:18:27.379 INFO:teuthology.orchestra.run.smithi084.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-29T11:18:27.379 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-03-29T11:18:27.380 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction Summary 2024-03-29T11:18:27.380 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2024-03-29T11:18:27.380 INFO:teuthology.orchestra.run.smithi084.stdout:Install 2 Packages 2024-03-29T11:18:27.380 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-03-29T11:18:27.380 INFO:teuthology.orchestra.run.smithi084.stdout:Total download size: 964 k 2024-03-29T11:18:27.380 INFO:teuthology.orchestra.run.smithi084.stdout:Installed size: 2.8 M 2024-03-29T11:18:27.380 INFO:teuthology.orchestra.run.smithi084.stdout:Downloading Packages: 2024-03-29T11:18:27.497 INFO:teuthology.orchestra.run.smithi084.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 672 kB/s | 79 kB 00:00 2024-03-29T11:18:27.661 INFO:teuthology.orchestra.run.smithi179.stdout: Running scriptlet: librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:18:27.946 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:18:27.946 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-29T11:18:27.946 INFO:teuthology.orchestra.run.smithi179.stdout:Installed: 2024-03-29T11:18:27.947 INFO:teuthology.orchestra.run.smithi179.stdout: librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:18:27.947 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-29T11:18:27.947 INFO:teuthology.orchestra.run.smithi179.stdout:Complete! 2024-03-29T11:18:28.201 DEBUG:teuthology.orchestra.run.smithi179:> sudo yum -y install libcephfs2 2024-03-29T11:18:28.247 INFO:teuthology.orchestra.run.smithi084.stdout:(2/2): ceph-fuse-19.0.0-1439.g5dcb190d.el9.x86_ 1.0 MB/s | 885 kB 00:00 2024-03-29T11:18:28.249 INFO:teuthology.orchestra.run.smithi084.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:18:28.249 INFO:teuthology.orchestra.run.smithi084.stdout:Total 1.1 MB/s | 964 kB 00:00 2024-03-29T11:18:28.249 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction check 2024-03-29T11:18:28.271 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction check succeeded. 2024-03-29T11:18:28.271 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction test 2024-03-29T11:18:28.345 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction test succeeded. 2024-03-29T11:18:28.345 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction 2024-03-29T11:18:28.534 INFO:teuthology.orchestra.run.smithi084.stdout: Preparing : 1/1 2024-03-29T11:18:28.657 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-03-29T11:18:28.731 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/2 2024-03-29T11:18:28.783 INFO:teuthology.orchestra.run.smithi179.stdout:Last metadata expiration check: 0:02:11 ago on Fri 29 Mar 2024 11:16:17 AM UTC. 2024-03-29T11:18:28.886 INFO:teuthology.orchestra.run.smithi179.stdout:Package libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:18:28.942 INFO:teuthology.orchestra.run.smithi179.stdout:Dependencies resolved. 2024-03-29T11:18:28.942 INFO:teuthology.orchestra.run.smithi179.stdout:Nothing to do. 2024-03-29T11:18:28.942 INFO:teuthology.orchestra.run.smithi179.stdout:Complete! 2024-03-29T11:18:29.012 DEBUG:teuthology.orchestra.run.smithi179:> sudo yum -y install libcephfs-devel 2024-03-29T11:18:29.437 INFO:teuthology.orchestra.run.smithi084.stdout: Running scriptlet: ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/2 2024-03-29T11:18:29.437 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/2 2024-03-29T11:18:29.581 INFO:teuthology.orchestra.run.smithi179.stdout:Last metadata expiration check: 0:02:12 ago on Fri 29 Mar 2024 11:16:17 AM UTC. 2024-03-29T11:18:29.733 INFO:teuthology.orchestra.run.smithi179.stdout:Dependencies resolved. 2024-03-29T11:18:29.734 INFO:teuthology.orchestra.run.smithi179.stdout:================================================================================ 2024-03-29T11:18:29.734 INFO:teuthology.orchestra.run.smithi179.stdout: Package Arch Version Repo Size 2024-03-29T11:18:29.734 INFO:teuthology.orchestra.run.smithi179.stdout:================================================================================ 2024-03-29T11:18:29.734 INFO:teuthology.orchestra.run.smithi179.stdout:Installing: 2024-03-29T11:18:29.734 INFO:teuthology.orchestra.run.smithi179.stdout: libcephfs-devel x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 31 k 2024-03-29T11:18:29.734 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-29T11:18:29.734 INFO:teuthology.orchestra.run.smithi179.stdout:Transaction Summary 2024-03-29T11:18:29.734 INFO:teuthology.orchestra.run.smithi179.stdout:================================================================================ 2024-03-29T11:18:29.734 INFO:teuthology.orchestra.run.smithi179.stdout:Install 1 Package 2024-03-29T11:18:29.734 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-29T11:18:29.734 INFO:teuthology.orchestra.run.smithi179.stdout:Total download size: 31 k 2024-03-29T11:18:29.734 INFO:teuthology.orchestra.run.smithi179.stdout:Installed size: 140 k 2024-03-29T11:18:29.734 INFO:teuthology.orchestra.run.smithi179.stdout:Downloading Packages: 2024-03-29T11:18:29.754 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-03-29T11:18:29.758 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-03-29T11:18:29.758 INFO:teuthology.orchestra.run.smithi084.stdout:Installed: 2024-03-29T11:18:29.758 INFO:teuthology.orchestra.run.smithi084.stdout: ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2024-03-29T11:18:29.758 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-03-29T11:18:29.758 INFO:teuthology.orchestra.run.smithi084.stdout:Complete! 2024-03-29T11:18:30.023 DEBUG:teuthology.orchestra.run.smithi084:> sudo yum -y install ceph-volume 2024-03-29T11:18:30.076 INFO:teuthology.orchestra.run.smithi179.stdout:libcephfs-devel-19.0.0-1439.g5dcb190d.el9.x86_6 91 kB/s | 31 kB 00:00 2024-03-29T11:18:30.077 INFO:teuthology.orchestra.run.smithi179.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:18:30.078 INFO:teuthology.orchestra.run.smithi179.stdout:Total 91 kB/s | 31 kB 00:00 2024-03-29T11:18:30.078 INFO:teuthology.orchestra.run.smithi179.stdout:Running transaction check 2024-03-29T11:18:30.081 INFO:teuthology.orchestra.run.smithi179.stdout:Transaction check succeeded. 2024-03-29T11:18:30.081 INFO:teuthology.orchestra.run.smithi179.stdout:Running transaction test 2024-03-29T11:18:30.108 INFO:teuthology.orchestra.run.smithi179.stdout:Transaction test succeeded. 2024-03-29T11:18:30.108 INFO:teuthology.orchestra.run.smithi179.stdout:Running transaction 2024-03-29T11:18:30.228 INFO:teuthology.orchestra.run.smithi179.stdout: Preparing : 1/1 2024-03-29T11:18:30.342 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:18:30.605 INFO:teuthology.orchestra.run.smithi084.stdout:Last metadata expiration check: 0:02:18 ago on Fri 29 Mar 2024 11:16:12 AM UTC. 2024-03-29T11:18:30.747 INFO:teuthology.orchestra.run.smithi179.stdout: Running scriptlet: libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:18:30.764 INFO:teuthology.orchestra.run.smithi084.stdout:Dependencies resolved. 2024-03-29T11:18:30.765 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2024-03-29T11:18:30.765 INFO:teuthology.orchestra.run.smithi084.stdout: Package Arch Version Repository Size 2024-03-29T11:18:30.765 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2024-03-29T11:18:30.765 INFO:teuthology.orchestra.run.smithi084.stdout:Installing: 2024-03-29T11:18:30.765 INFO:teuthology.orchestra.run.smithi084.stdout: ceph-volume noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 264 k 2024-03-29T11:18:30.765 INFO:teuthology.orchestra.run.smithi084.stdout:Installing dependencies: 2024-03-29T11:18:30.765 INFO:teuthology.orchestra.run.smithi084.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-29T11:18:30.765 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-03-29T11:18:30.765 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction Summary 2024-03-29T11:18:30.765 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2024-03-29T11:18:30.766 INFO:teuthology.orchestra.run.smithi084.stdout:Install 2 Packages 2024-03-29T11:18:30.766 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-03-29T11:18:30.766 INFO:teuthology.orchestra.run.smithi084.stdout:Total download size: 332 k 2024-03-29T11:18:30.766 INFO:teuthology.orchestra.run.smithi084.stdout:Installed size: 1.5 M 2024-03-29T11:18:30.766 INFO:teuthology.orchestra.run.smithi084.stdout:Downloading Packages: 2024-03-29T11:18:30.901 INFO:teuthology.orchestra.run.smithi084.stdout:(1/2): python3-packaging-20.9-5.el9.noarch.rpm 513 kB/s | 69 kB 00:00 2024-03-29T11:18:31.011 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:18:31.011 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-29T11:18:31.011 INFO:teuthology.orchestra.run.smithi179.stdout:Installed: 2024-03-29T11:18:31.011 INFO:teuthology.orchestra.run.smithi179.stdout: libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:18:31.011 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-29T11:18:31.011 INFO:teuthology.orchestra.run.smithi179.stdout:Complete! 2024-03-29T11:18:31.067 INFO:teuthology.orchestra.run.smithi084.stdout:(2/2): ceph-volume-19.0.0-1439.g5dcb190d.el9.no 877 kB/s | 264 kB 00:00 2024-03-29T11:18:31.068 INFO:teuthology.orchestra.run.smithi084.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:18:31.069 INFO:teuthology.orchestra.run.smithi084.stdout:Total 1.1 MB/s | 332 kB 00:00 2024-03-29T11:18:31.069 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction check 2024-03-29T11:18:31.077 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction check succeeded. 2024-03-29T11:18:31.077 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction test 2024-03-29T11:18:31.113 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction test succeeded. 2024-03-29T11:18:31.114 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction 2024-03-29T11:18:31.171 DEBUG:teuthology.orchestra.run.smithi179:> sudo yum -y install librados2 2024-03-29T11:18:31.259 INFO:teuthology.orchestra.run.smithi084.stdout: Preparing : 1/1 2024-03-29T11:18:31.433 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-packaging-20.9-5.el9.noarch 1/2 2024-03-29T11:18:31.478 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2/2 2024-03-29T11:18:31.494 INFO:teuthology.orchestra.run.smithi084.stdout: Running scriptlet: ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2/2 2024-03-29T11:18:31.494 INFO:teuthology.orchestra.run.smithi084.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T11:18:31.494 INFO:teuthology.orchestra.run.smithi084.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-29T11:18:31.494 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-03-29T11:18:31.733 INFO:teuthology.orchestra.run.smithi179.stdout:Last metadata expiration check: 0:02:14 ago on Fri 29 Mar 2024 11:16:17 AM UTC. 2024-03-29T11:18:31.830 INFO:teuthology.orchestra.run.smithi179.stdout:Package librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:18:31.881 INFO:teuthology.orchestra.run.smithi179.stdout:Dependencies resolved. 2024-03-29T11:18:31.882 INFO:teuthology.orchestra.run.smithi179.stdout:Nothing to do. 2024-03-29T11:18:31.882 INFO:teuthology.orchestra.run.smithi179.stdout:Complete! 2024-03-29T11:18:31.935 DEBUG:teuthology.orchestra.run.smithi179:> sudo yum -y install librbd1 2024-03-29T11:18:32.179 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 1/2 2024-03-29T11:18:32.501 INFO:teuthology.orchestra.run.smithi179.stdout:Last metadata expiration check: 0:02:15 ago on Fri 29 Mar 2024 11:16:17 AM UTC. 2024-03-29T11:18:32.513 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 2/2 2024-03-29T11:18:32.514 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-03-29T11:18:32.514 INFO:teuthology.orchestra.run.smithi084.stdout:Installed: 2024-03-29T11:18:32.514 INFO:teuthology.orchestra.run.smithi084.stdout: ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:18:32.514 INFO:teuthology.orchestra.run.smithi084.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-29T11:18:32.514 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-03-29T11:18:32.514 INFO:teuthology.orchestra.run.smithi084.stdout:Complete! 2024-03-29T11:18:32.599 INFO:teuthology.orchestra.run.smithi179.stdout:Package librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:18:32.651 INFO:teuthology.orchestra.run.smithi179.stdout:Dependencies resolved. 2024-03-29T11:18:32.652 INFO:teuthology.orchestra.run.smithi179.stdout:Nothing to do. 2024-03-29T11:18:32.652 INFO:teuthology.orchestra.run.smithi179.stdout:Complete! 2024-03-29T11:18:32.698 DEBUG:teuthology.orchestra.run.smithi179:> sudo yum -y install python3-rados 2024-03-29T11:18:32.851 DEBUG:teuthology.orchestra.run.smithi084:> sudo yum -y install librados-devel 2024-03-29T11:18:33.265 INFO:teuthology.orchestra.run.smithi179.stdout:Last metadata expiration check: 0:02:16 ago on Fri 29 Mar 2024 11:16:17 AM UTC. 2024-03-29T11:18:33.363 INFO:teuthology.orchestra.run.smithi179.stdout:Package python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:18:33.415 INFO:teuthology.orchestra.run.smithi179.stdout:Dependencies resolved. 2024-03-29T11:18:33.416 INFO:teuthology.orchestra.run.smithi179.stdout:Nothing to do. 2024-03-29T11:18:33.416 INFO:teuthology.orchestra.run.smithi179.stdout:Complete! 2024-03-29T11:18:33.431 INFO:teuthology.orchestra.run.smithi084.stdout:Last metadata expiration check: 0:02:21 ago on Fri 29 Mar 2024 11:16:12 AM UTC. 2024-03-29T11:18:33.462 DEBUG:teuthology.orchestra.run.smithi179:> sudo yum -y install python3-rgw 2024-03-29T11:18:33.582 INFO:teuthology.orchestra.run.smithi084.stdout:Dependencies resolved. 2024-03-29T11:18:33.582 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2024-03-29T11:18:33.582 INFO:teuthology.orchestra.run.smithi084.stdout: Package Arch Version Repo Size 2024-03-29T11:18:33.582 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2024-03-29T11:18:33.582 INFO:teuthology.orchestra.run.smithi084.stdout:Installing: 2024-03-29T11:18:33.583 INFO:teuthology.orchestra.run.smithi084.stdout: librados-devel x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 127 k 2024-03-29T11:18:33.583 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-03-29T11:18:33.583 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction Summary 2024-03-29T11:18:33.583 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2024-03-29T11:18:33.583 INFO:teuthology.orchestra.run.smithi084.stdout:Install 1 Package 2024-03-29T11:18:33.583 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-03-29T11:18:33.583 INFO:teuthology.orchestra.run.smithi084.stdout:Total download size: 127 k 2024-03-29T11:18:33.584 INFO:teuthology.orchestra.run.smithi084.stdout:Installed size: 456 k 2024-03-29T11:18:33.584 INFO:teuthology.orchestra.run.smithi084.stdout:Downloading Packages: 2024-03-29T11:18:33.989 INFO:teuthology.orchestra.run.smithi084.stdout:librados-devel-19.0.0-1439.g5dcb190d.el9.x86_64 312 kB/s | 127 kB 00:00 2024-03-29T11:18:33.990 INFO:teuthology.orchestra.run.smithi084.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:18:33.990 INFO:teuthology.orchestra.run.smithi084.stdout:Total 311 kB/s | 127 kB 00:00 2024-03-29T11:18:33.990 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction check 2024-03-29T11:18:33.996 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction check succeeded. 2024-03-29T11:18:33.996 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction test 2024-03-29T11:18:34.029 INFO:teuthology.orchestra.run.smithi179.stdout:Last metadata expiration check: 0:02:17 ago on Fri 29 Mar 2024 11:16:17 AM UTC. 2024-03-29T11:18:34.053 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction test succeeded. 2024-03-29T11:18:34.053 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction 2024-03-29T11:18:34.130 INFO:teuthology.orchestra.run.smithi179.stdout:Package python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:18:34.182 INFO:teuthology.orchestra.run.smithi179.stdout:Dependencies resolved. 2024-03-29T11:18:34.183 INFO:teuthology.orchestra.run.smithi179.stdout:Nothing to do. 2024-03-29T11:18:34.183 INFO:teuthology.orchestra.run.smithi179.stdout:Complete! 2024-03-29T11:18:34.205 INFO:teuthology.orchestra.run.smithi084.stdout: Preparing : 1/1 2024-03-29T11:18:34.228 DEBUG:teuthology.orchestra.run.smithi179:> sudo yum -y install python3-cephfs 2024-03-29T11:18:34.313 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:18:34.744 INFO:teuthology.orchestra.run.smithi084.stdout: Running scriptlet: librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:18:34.802 INFO:teuthology.orchestra.run.smithi179.stdout:Last metadata expiration check: 0:02:17 ago on Fri 29 Mar 2024 11:16:17 AM UTC. 2024-03-29T11:18:34.901 INFO:teuthology.orchestra.run.smithi179.stdout:Package python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:18:34.952 INFO:teuthology.orchestra.run.smithi179.stdout:Dependencies resolved. 2024-03-29T11:18:34.953 INFO:teuthology.orchestra.run.smithi179.stdout:Nothing to do. 2024-03-29T11:18:34.953 INFO:teuthology.orchestra.run.smithi179.stdout:Complete! 2024-03-29T11:18:34.995 DEBUG:teuthology.orchestra.run.smithi179:> sudo yum -y install python3-rbd 2024-03-29T11:18:35.016 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:18:35.016 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-03-29T11:18:35.016 INFO:teuthology.orchestra.run.smithi084.stdout:Installed: 2024-03-29T11:18:35.016 INFO:teuthology.orchestra.run.smithi084.stdout: librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:18:35.016 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-03-29T11:18:35.016 INFO:teuthology.orchestra.run.smithi084.stdout:Complete! 2024-03-29T11:18:35.266 DEBUG:teuthology.orchestra.run.smithi084:> sudo yum -y install libcephfs2 2024-03-29T11:18:35.559 INFO:teuthology.orchestra.run.smithi179.stdout:Last metadata expiration check: 0:02:18 ago on Fri 29 Mar 2024 11:16:17 AM UTC. 2024-03-29T11:18:35.659 INFO:teuthology.orchestra.run.smithi179.stdout:Package python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:18:35.711 INFO:teuthology.orchestra.run.smithi179.stdout:Dependencies resolved. 2024-03-29T11:18:35.712 INFO:teuthology.orchestra.run.smithi179.stdout:Nothing to do. 2024-03-29T11:18:35.712 INFO:teuthology.orchestra.run.smithi179.stdout:Complete! 2024-03-29T11:18:35.757 DEBUG:teuthology.orchestra.run.smithi179:> sudo yum -y install rbd-fuse 2024-03-29T11:18:35.845 INFO:teuthology.orchestra.run.smithi084.stdout:Last metadata expiration check: 0:02:23 ago on Fri 29 Mar 2024 11:16:12 AM UTC. 2024-03-29T11:18:35.946 INFO:teuthology.orchestra.run.smithi084.stdout:Package libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:18:36.000 INFO:teuthology.orchestra.run.smithi084.stdout:Dependencies resolved. 2024-03-29T11:18:36.001 INFO:teuthology.orchestra.run.smithi084.stdout:Nothing to do. 2024-03-29T11:18:36.001 INFO:teuthology.orchestra.run.smithi084.stdout:Complete! 2024-03-29T11:18:36.062 DEBUG:teuthology.orchestra.run.smithi084:> sudo yum -y install libcephfs-devel 2024-03-29T11:18:36.319 INFO:teuthology.orchestra.run.smithi179.stdout:Last metadata expiration check: 0:02:19 ago on Fri 29 Mar 2024 11:16:17 AM UTC. 2024-03-29T11:18:36.472 INFO:teuthology.orchestra.run.smithi179.stdout:Dependencies resolved. 2024-03-29T11:18:36.473 INFO:teuthology.orchestra.run.smithi179.stdout:================================================================================ 2024-03-29T11:18:36.473 INFO:teuthology.orchestra.run.smithi179.stdout: Package Architecture Version Repository Size 2024-03-29T11:18:36.473 INFO:teuthology.orchestra.run.smithi179.stdout:================================================================================ 2024-03-29T11:18:36.473 INFO:teuthology.orchestra.run.smithi179.stdout:Installing: 2024-03-29T11:18:36.473 INFO:teuthology.orchestra.run.smithi179.stdout: rbd-fuse x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 85 k 2024-03-29T11:18:36.473 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-29T11:18:36.473 INFO:teuthology.orchestra.run.smithi179.stdout:Transaction Summary 2024-03-29T11:18:36.474 INFO:teuthology.orchestra.run.smithi179.stdout:================================================================================ 2024-03-29T11:18:36.474 INFO:teuthology.orchestra.run.smithi179.stdout:Install 1 Package 2024-03-29T11:18:36.474 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-29T11:18:36.474 INFO:teuthology.orchestra.run.smithi179.stdout:Total download size: 85 k 2024-03-29T11:18:36.474 INFO:teuthology.orchestra.run.smithi179.stdout:Installed size: 231 k 2024-03-29T11:18:36.474 INFO:teuthology.orchestra.run.smithi179.stdout:Downloading Packages: 2024-03-29T11:18:36.638 INFO:teuthology.orchestra.run.smithi084.stdout:Last metadata expiration check: 0:02:24 ago on Fri 29 Mar 2024 11:16:12 AM UTC. 2024-03-29T11:18:36.790 INFO:teuthology.orchestra.run.smithi084.stdout:Dependencies resolved. 2024-03-29T11:18:36.791 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2024-03-29T11:18:36.791 INFO:teuthology.orchestra.run.smithi084.stdout: Package Arch Version Repo Size 2024-03-29T11:18:36.791 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2024-03-29T11:18:36.791 INFO:teuthology.orchestra.run.smithi084.stdout:Installing: 2024-03-29T11:18:36.791 INFO:teuthology.orchestra.run.smithi084.stdout: libcephfs-devel x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 31 k 2024-03-29T11:18:36.791 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-03-29T11:18:36.791 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction Summary 2024-03-29T11:18:36.791 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2024-03-29T11:18:36.791 INFO:teuthology.orchestra.run.smithi084.stdout:Install 1 Package 2024-03-29T11:18:36.791 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-03-29T11:18:36.791 INFO:teuthology.orchestra.run.smithi084.stdout:Total download size: 31 k 2024-03-29T11:18:36.791 INFO:teuthology.orchestra.run.smithi084.stdout:Installed size: 140 k 2024-03-29T11:18:36.792 INFO:teuthology.orchestra.run.smithi084.stdout:Downloading Packages: 2024-03-29T11:18:36.804 INFO:teuthology.orchestra.run.smithi179.stdout:rbd-fuse-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 258 kB/s | 85 kB 00:00 2024-03-29T11:18:36.805 INFO:teuthology.orchestra.run.smithi179.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:18:36.805 INFO:teuthology.orchestra.run.smithi179.stdout:Total 257 kB/s | 85 kB 00:00 2024-03-29T11:18:36.806 INFO:teuthology.orchestra.run.smithi179.stdout:Running transaction check 2024-03-29T11:18:36.813 INFO:teuthology.orchestra.run.smithi179.stdout:Transaction check succeeded. 2024-03-29T11:18:36.813 INFO:teuthology.orchestra.run.smithi179.stdout:Running transaction test 2024-03-29T11:18:36.868 INFO:teuthology.orchestra.run.smithi179.stdout:Transaction test succeeded. 2024-03-29T11:18:36.869 INFO:teuthology.orchestra.run.smithi179.stdout:Running transaction 2024-03-29T11:18:36.975 INFO:teuthology.orchestra.run.smithi084.stdout:libcephfs-devel-19.0.0-1439.g5dcb190d.el9.x86_6 169 kB/s | 31 kB 00:00 2024-03-29T11:18:36.976 INFO:teuthology.orchestra.run.smithi084.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:18:36.976 INFO:teuthology.orchestra.run.smithi084.stdout:Total 168 kB/s | 31 kB 00:00 2024-03-29T11:18:36.976 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction check 2024-03-29T11:18:36.980 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction check succeeded. 2024-03-29T11:18:36.980 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction test 2024-03-29T11:18:37.004 INFO:teuthology.orchestra.run.smithi179.stdout: Preparing : 1/1 2024-03-29T11:18:37.005 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction test succeeded. 2024-03-29T11:18:37.005 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction 2024-03-29T11:18:37.103 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:18:37.155 INFO:teuthology.orchestra.run.smithi084.stdout: Preparing : 1/1 2024-03-29T11:18:37.233 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:18:37.524 INFO:teuthology.orchestra.run.smithi179.stdout: Running scriptlet: rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:18:37.597 INFO:teuthology.orchestra.run.smithi084.stdout: Running scriptlet: libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:18:37.828 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:18:37.828 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-29T11:18:37.828 INFO:teuthology.orchestra.run.smithi179.stdout:Installed: 2024-03-29T11:18:37.828 INFO:teuthology.orchestra.run.smithi179.stdout: rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:18:37.828 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-29T11:18:37.828 INFO:teuthology.orchestra.run.smithi179.stdout:Complete! 2024-03-29T11:18:37.861 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:18:37.861 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-03-29T11:18:37.861 INFO:teuthology.orchestra.run.smithi084.stdout:Installed: 2024-03-29T11:18:37.861 INFO:teuthology.orchestra.run.smithi084.stdout: libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:18:37.862 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-03-29T11:18:37.862 INFO:teuthology.orchestra.run.smithi084.stdout:Complete! 2024-03-29T11:18:38.027 DEBUG:teuthology.orchestra.run.smithi084:> sudo yum -y install librados2 2024-03-29T11:18:38.049 DEBUG:teuthology.orchestra.run.smithi179:> sudo yum -y install rbd-mirror 2024-03-29T11:18:38.585 INFO:teuthology.orchestra.run.smithi084.stdout:Last metadata expiration check: 0:02:26 ago on Fri 29 Mar 2024 11:16:12 AM UTC. 2024-03-29T11:18:38.628 INFO:teuthology.orchestra.run.smithi179.stdout:Last metadata expiration check: 0:02:21 ago on Fri 29 Mar 2024 11:16:17 AM UTC. 2024-03-29T11:18:38.682 INFO:teuthology.orchestra.run.smithi084.stdout:Package librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:18:38.734 INFO:teuthology.orchestra.run.smithi084.stdout:Dependencies resolved. 2024-03-29T11:18:38.734 INFO:teuthology.orchestra.run.smithi084.stdout:Nothing to do. 2024-03-29T11:18:38.734 INFO:teuthology.orchestra.run.smithi084.stdout:Complete! 2024-03-29T11:18:38.777 DEBUG:teuthology.orchestra.run.smithi084:> sudo yum -y install librbd1 2024-03-29T11:18:38.785 INFO:teuthology.orchestra.run.smithi179.stdout:Dependencies resolved. 2024-03-29T11:18:38.786 INFO:teuthology.orchestra.run.smithi179.stdout:================================================================================ 2024-03-29T11:18:38.786 INFO:teuthology.orchestra.run.smithi179.stdout: Package Arch Version Repo Size 2024-03-29T11:18:38.786 INFO:teuthology.orchestra.run.smithi179.stdout:================================================================================ 2024-03-29T11:18:38.786 INFO:teuthology.orchestra.run.smithi179.stdout:Installing: 2024-03-29T11:18:38.786 INFO:teuthology.orchestra.run.smithi179.stdout: rbd-mirror x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.1 M 2024-03-29T11:18:38.786 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-29T11:18:38.786 INFO:teuthology.orchestra.run.smithi179.stdout:Transaction Summary 2024-03-29T11:18:38.786 INFO:teuthology.orchestra.run.smithi179.stdout:================================================================================ 2024-03-29T11:18:38.787 INFO:teuthology.orchestra.run.smithi179.stdout:Install 1 Package 2024-03-29T11:18:38.787 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-29T11:18:38.787 INFO:teuthology.orchestra.run.smithi179.stdout:Total download size: 3.1 M 2024-03-29T11:18:38.787 INFO:teuthology.orchestra.run.smithi179.stdout:Installed size: 13 M 2024-03-29T11:18:38.787 INFO:teuthology.orchestra.run.smithi179.stdout:Downloading Packages: 2024-03-29T11:18:39.338 INFO:teuthology.orchestra.run.smithi084.stdout:Last metadata expiration check: 0:02:27 ago on Fri 29 Mar 2024 11:16:12 AM UTC. 2024-03-29T11:18:39.435 INFO:teuthology.orchestra.run.smithi084.stdout:Package librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:18:39.488 INFO:teuthology.orchestra.run.smithi084.stdout:Dependencies resolved. 2024-03-29T11:18:39.489 INFO:teuthology.orchestra.run.smithi084.stdout:Nothing to do. 2024-03-29T11:18:39.489 INFO:teuthology.orchestra.run.smithi084.stdout:Complete! 2024-03-29T11:18:39.516 INFO:teuthology.orchestra.run.smithi179.stdout:rbd-mirror-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 4.2 MB/s | 3.1 MB 00:00 2024-03-29T11:18:39.517 INFO:teuthology.orchestra.run.smithi179.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:18:39.517 INFO:teuthology.orchestra.run.smithi179.stdout:Total 4.2 MB/s | 3.1 MB 00:00 2024-03-29T11:18:39.517 INFO:teuthology.orchestra.run.smithi179.stdout:Running transaction check 2024-03-29T11:18:39.526 INFO:teuthology.orchestra.run.smithi179.stdout:Transaction check succeeded. 2024-03-29T11:18:39.526 INFO:teuthology.orchestra.run.smithi179.stdout:Running transaction test 2024-03-29T11:18:39.533 DEBUG:teuthology.orchestra.run.smithi084:> sudo yum -y install python3-rados 2024-03-29T11:18:39.599 INFO:teuthology.orchestra.run.smithi179.stdout:Transaction test succeeded. 2024-03-29T11:18:39.599 INFO:teuthology.orchestra.run.smithi179.stdout:Running transaction 2024-03-29T11:18:40.040 INFO:teuthology.orchestra.run.smithi179.stdout: Preparing : 1/1 2024-03-29T11:18:40.106 INFO:teuthology.orchestra.run.smithi084.stdout:Last metadata expiration check: 0:02:28 ago on Fri 29 Mar 2024 11:16:12 AM UTC. 2024-03-29T11:18:40.108 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:18:40.132 INFO:teuthology.orchestra.run.smithi179.stdout: Running scriptlet: rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:18:40.133 INFO:teuthology.orchestra.run.smithi179.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T11:18:40.133 INFO:teuthology.orchestra.run.smithi179.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-29T11:18:40.133 INFO:teuthology.orchestra.run.smithi179.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-29T11:18:40.133 INFO:teuthology.orchestra.run.smithi179.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-29T11:18:40.133 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-29T11:18:40.205 INFO:teuthology.orchestra.run.smithi084.stdout:Package python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:18:40.256 INFO:teuthology.orchestra.run.smithi084.stdout:Dependencies resolved. 2024-03-29T11:18:40.257 INFO:teuthology.orchestra.run.smithi084.stdout:Nothing to do. 2024-03-29T11:18:40.258 INFO:teuthology.orchestra.run.smithi084.stdout:Complete! 2024-03-29T11:18:40.308 DEBUG:teuthology.orchestra.run.smithi084:> sudo yum -y install python3-rgw 2024-03-29T11:18:40.888 INFO:teuthology.orchestra.run.smithi084.stdout:Last metadata expiration check: 0:02:28 ago on Fri 29 Mar 2024 11:16:12 AM UTC. 2024-03-29T11:18:40.988 INFO:teuthology.orchestra.run.smithi084.stdout:Package python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:18:41.040 INFO:teuthology.orchestra.run.smithi084.stdout:Dependencies resolved. 2024-03-29T11:18:41.041 INFO:teuthology.orchestra.run.smithi084.stdout:Nothing to do. 2024-03-29T11:18:41.041 INFO:teuthology.orchestra.run.smithi084.stdout:Complete! 2024-03-29T11:18:41.098 DEBUG:teuthology.orchestra.run.smithi084:> sudo yum -y install python3-cephfs 2024-03-29T11:18:41.260 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:18:41.260 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-29T11:18:41.260 INFO:teuthology.orchestra.run.smithi179.stdout:Installed: 2024-03-29T11:18:41.260 INFO:teuthology.orchestra.run.smithi179.stdout: rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:18:41.260 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-29T11:18:41.261 INFO:teuthology.orchestra.run.smithi179.stdout:Complete! 2024-03-29T11:18:41.505 DEBUG:teuthology.orchestra.run.smithi179:> sudo yum -y install rbd-nbd 2024-03-29T11:18:41.678 INFO:teuthology.orchestra.run.smithi084.stdout:Last metadata expiration check: 0:02:29 ago on Fri 29 Mar 2024 11:16:12 AM UTC. 2024-03-29T11:18:41.775 INFO:teuthology.orchestra.run.smithi084.stdout:Package python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:18:41.827 INFO:teuthology.orchestra.run.smithi084.stdout:Dependencies resolved. 2024-03-29T11:18:41.828 INFO:teuthology.orchestra.run.smithi084.stdout:Nothing to do. 2024-03-29T11:18:41.828 INFO:teuthology.orchestra.run.smithi084.stdout:Complete! 2024-03-29T11:18:41.873 DEBUG:teuthology.orchestra.run.smithi084:> sudo yum -y install python3-rbd 2024-03-29T11:18:42.082 INFO:teuthology.orchestra.run.smithi179.stdout:Last metadata expiration check: 0:02:25 ago on Fri 29 Mar 2024 11:16:17 AM UTC. 2024-03-29T11:18:42.233 INFO:teuthology.orchestra.run.smithi179.stdout:Dependencies resolved. 2024-03-29T11:18:42.233 INFO:teuthology.orchestra.run.smithi179.stdout:================================================================================ 2024-03-29T11:18:42.233 INFO:teuthology.orchestra.run.smithi179.stdout: Package Architecture Version Repository Size 2024-03-29T11:18:42.233 INFO:teuthology.orchestra.run.smithi179.stdout:================================================================================ 2024-03-29T11:18:42.234 INFO:teuthology.orchestra.run.smithi179.stdout:Installing: 2024-03-29T11:18:42.234 INFO:teuthology.orchestra.run.smithi179.stdout: rbd-nbd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 171 k 2024-03-29T11:18:42.234 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-29T11:18:42.234 INFO:teuthology.orchestra.run.smithi179.stdout:Transaction Summary 2024-03-29T11:18:42.234 INFO:teuthology.orchestra.run.smithi179.stdout:================================================================================ 2024-03-29T11:18:42.234 INFO:teuthology.orchestra.run.smithi179.stdout:Install 1 Package 2024-03-29T11:18:42.234 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-29T11:18:42.234 INFO:teuthology.orchestra.run.smithi179.stdout:Total download size: 171 k 2024-03-29T11:18:42.234 INFO:teuthology.orchestra.run.smithi179.stdout:Installed size: 494 k 2024-03-29T11:18:42.234 INFO:teuthology.orchestra.run.smithi179.stdout:Downloading Packages: 2024-03-29T11:18:42.434 INFO:teuthology.orchestra.run.smithi084.stdout:Last metadata expiration check: 0:02:30 ago on Fri 29 Mar 2024 11:16:12 AM UTC. 2024-03-29T11:18:42.470 INFO:teuthology.orchestra.run.smithi179.stdout:rbd-nbd-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 729 kB/s | 171 kB 00:00 2024-03-29T11:18:42.471 INFO:teuthology.orchestra.run.smithi179.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:18:42.471 INFO:teuthology.orchestra.run.smithi179.stdout:Total 724 kB/s | 171 kB 00:00 2024-03-29T11:18:42.471 INFO:teuthology.orchestra.run.smithi179.stdout:Running transaction check 2024-03-29T11:18:42.478 INFO:teuthology.orchestra.run.smithi179.stdout:Transaction check succeeded. 2024-03-29T11:18:42.478 INFO:teuthology.orchestra.run.smithi179.stdout:Running transaction test 2024-03-29T11:18:42.531 INFO:teuthology.orchestra.run.smithi084.stdout:Package python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:18:42.534 INFO:teuthology.orchestra.run.smithi179.stdout:Transaction test succeeded. 2024-03-29T11:18:42.534 INFO:teuthology.orchestra.run.smithi179.stdout:Running transaction 2024-03-29T11:18:42.583 INFO:teuthology.orchestra.run.smithi084.stdout:Dependencies resolved. 2024-03-29T11:18:42.584 INFO:teuthology.orchestra.run.smithi084.stdout:Nothing to do. 2024-03-29T11:18:42.584 INFO:teuthology.orchestra.run.smithi084.stdout:Complete! 2024-03-29T11:18:42.629 DEBUG:teuthology.orchestra.run.smithi084:> sudo yum -y install rbd-fuse 2024-03-29T11:18:42.680 INFO:teuthology.orchestra.run.smithi179.stdout: Preparing : 1/1 2024-03-29T11:18:42.785 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:18:43.184 INFO:teuthology.orchestra.run.smithi084.stdout:Last metadata expiration check: 0:02:31 ago on Fri 29 Mar 2024 11:16:12 AM UTC. 2024-03-29T11:18:43.202 INFO:teuthology.orchestra.run.smithi179.stdout: Running scriptlet: rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:18:43.332 INFO:teuthology.orchestra.run.smithi084.stdout:Dependencies resolved. 2024-03-29T11:18:43.332 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2024-03-29T11:18:43.332 INFO:teuthology.orchestra.run.smithi084.stdout: Package Architecture Version Repository Size 2024-03-29T11:18:43.333 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2024-03-29T11:18:43.333 INFO:teuthology.orchestra.run.smithi084.stdout:Installing: 2024-03-29T11:18:43.333 INFO:teuthology.orchestra.run.smithi084.stdout: rbd-fuse x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 85 k 2024-03-29T11:18:43.333 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-03-29T11:18:43.333 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction Summary 2024-03-29T11:18:43.333 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2024-03-29T11:18:43.333 INFO:teuthology.orchestra.run.smithi084.stdout:Install 1 Package 2024-03-29T11:18:43.333 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-03-29T11:18:43.333 INFO:teuthology.orchestra.run.smithi084.stdout:Total download size: 85 k 2024-03-29T11:18:43.333 INFO:teuthology.orchestra.run.smithi084.stdout:Installed size: 231 k 2024-03-29T11:18:43.333 INFO:teuthology.orchestra.run.smithi084.stdout:Downloading Packages: 2024-03-29T11:18:43.467 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:18:43.467 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-29T11:18:43.468 INFO:teuthology.orchestra.run.smithi179.stdout:Installed: 2024-03-29T11:18:43.468 INFO:teuthology.orchestra.run.smithi179.stdout: rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:18:43.468 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-29T11:18:43.468 INFO:teuthology.orchestra.run.smithi179.stdout:Complete! 2024-03-29T11:18:43.536 INFO:teuthology.orchestra.run.smithi084.stdout:rbd-fuse-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 421 kB/s | 85 kB 00:00 2024-03-29T11:18:43.536 INFO:teuthology.orchestra.run.smithi084.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:18:43.537 INFO:teuthology.orchestra.run.smithi084.stdout:Total 418 kB/s | 85 kB 00:00 2024-03-29T11:18:43.537 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction check 2024-03-29T11:18:43.544 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction check succeeded. 2024-03-29T11:18:43.544 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction test 2024-03-29T11:18:43.598 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction test succeeded. 2024-03-29T11:18:43.599 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction 2024-03-29T11:18:43.715 DEBUG:teuthology.parallel:result is None 2024-03-29T11:18:43.735 INFO:teuthology.orchestra.run.smithi084.stdout: Preparing : 1/1 2024-03-29T11:18:43.836 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:18:44.238 INFO:teuthology.orchestra.run.smithi084.stdout: Running scriptlet: rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:18:44.526 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:18:44.526 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-03-29T11:18:44.526 INFO:teuthology.orchestra.run.smithi084.stdout:Installed: 2024-03-29T11:18:44.527 INFO:teuthology.orchestra.run.smithi084.stdout: rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:18:44.527 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-03-29T11:18:44.527 INFO:teuthology.orchestra.run.smithi084.stdout:Complete! 2024-03-29T11:18:44.771 DEBUG:teuthology.orchestra.run.smithi084:> sudo yum -y install rbd-mirror 2024-03-29T11:18:45.346 INFO:teuthology.orchestra.run.smithi084.stdout:Last metadata expiration check: 0:02:33 ago on Fri 29 Mar 2024 11:16:12 AM UTC. 2024-03-29T11:18:45.502 INFO:teuthology.orchestra.run.smithi084.stdout:Dependencies resolved. 2024-03-29T11:18:45.503 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2024-03-29T11:18:45.503 INFO:teuthology.orchestra.run.smithi084.stdout: Package Arch Version Repo Size 2024-03-29T11:18:45.503 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2024-03-29T11:18:45.503 INFO:teuthology.orchestra.run.smithi084.stdout:Installing: 2024-03-29T11:18:45.503 INFO:teuthology.orchestra.run.smithi084.stdout: rbd-mirror x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.1 M 2024-03-29T11:18:45.503 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-03-29T11:18:45.503 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction Summary 2024-03-29T11:18:45.503 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2024-03-29T11:18:45.503 INFO:teuthology.orchestra.run.smithi084.stdout:Install 1 Package 2024-03-29T11:18:45.503 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-03-29T11:18:45.503 INFO:teuthology.orchestra.run.smithi084.stdout:Total download size: 3.1 M 2024-03-29T11:18:45.503 INFO:teuthology.orchestra.run.smithi084.stdout:Installed size: 13 M 2024-03-29T11:18:45.503 INFO:teuthology.orchestra.run.smithi084.stdout:Downloading Packages: 2024-03-29T11:18:45.980 INFO:teuthology.orchestra.run.smithi084.stdout:rbd-mirror-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 6.5 MB/s | 3.1 MB 00:00 2024-03-29T11:18:45.981 INFO:teuthology.orchestra.run.smithi084.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:18:45.982 INFO:teuthology.orchestra.run.smithi084.stdout:Total 6.5 MB/s | 3.1 MB 00:00 2024-03-29T11:18:45.982 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction check 2024-03-29T11:18:45.990 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction check succeeded. 2024-03-29T11:18:45.990 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction test 2024-03-29T11:18:46.063 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction test succeeded. 2024-03-29T11:18:46.063 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction 2024-03-29T11:18:46.488 INFO:teuthology.orchestra.run.smithi084.stdout: Preparing : 1/1 2024-03-29T11:18:46.548 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:18:46.572 INFO:teuthology.orchestra.run.smithi084.stdout: Running scriptlet: rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:18:46.572 INFO:teuthology.orchestra.run.smithi084.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T11:18:46.572 INFO:teuthology.orchestra.run.smithi084.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-29T11:18:46.573 INFO:teuthology.orchestra.run.smithi084.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-29T11:18:46.573 INFO:teuthology.orchestra.run.smithi084.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-29T11:18:46.573 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-03-29T11:18:47.726 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:18:47.726 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-03-29T11:18:47.726 INFO:teuthology.orchestra.run.smithi084.stdout:Installed: 2024-03-29T11:18:47.726 INFO:teuthology.orchestra.run.smithi084.stdout: rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:18:47.726 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-03-29T11:18:47.726 INFO:teuthology.orchestra.run.smithi084.stdout:Complete! 2024-03-29T11:18:47.978 DEBUG:teuthology.orchestra.run.smithi084:> sudo yum -y install rbd-nbd 2024-03-29T11:18:48.551 INFO:teuthology.orchestra.run.smithi084.stdout:Last metadata expiration check: 0:02:36 ago on Fri 29 Mar 2024 11:16:12 AM UTC. 2024-03-29T11:18:48.703 INFO:teuthology.orchestra.run.smithi084.stdout:Dependencies resolved. 2024-03-29T11:18:48.704 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2024-03-29T11:18:48.705 INFO:teuthology.orchestra.run.smithi084.stdout: Package Architecture Version Repository Size 2024-03-29T11:18:48.705 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2024-03-29T11:18:48.705 INFO:teuthology.orchestra.run.smithi084.stdout:Installing: 2024-03-29T11:18:48.705 INFO:teuthology.orchestra.run.smithi084.stdout: rbd-nbd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 171 k 2024-03-29T11:18:48.705 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-03-29T11:18:48.706 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction Summary 2024-03-29T11:18:48.706 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2024-03-29T11:18:48.706 INFO:teuthology.orchestra.run.smithi084.stdout:Install 1 Package 2024-03-29T11:18:48.706 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-03-29T11:18:48.706 INFO:teuthology.orchestra.run.smithi084.stdout:Total download size: 171 k 2024-03-29T11:18:48.706 INFO:teuthology.orchestra.run.smithi084.stdout:Installed size: 494 k 2024-03-29T11:18:48.707 INFO:teuthology.orchestra.run.smithi084.stdout:Downloading Packages: 2024-03-29T11:18:49.358 INFO:teuthology.orchestra.run.smithi084.stdout:rbd-nbd-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 262 kB/s | 171 kB 00:00 2024-03-29T11:18:49.358 INFO:teuthology.orchestra.run.smithi084.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:18:49.358 INFO:teuthology.orchestra.run.smithi084.stdout:Total 262 kB/s | 171 kB 00:00 2024-03-29T11:18:49.359 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction check 2024-03-29T11:18:49.367 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction check succeeded. 2024-03-29T11:18:49.367 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction test 2024-03-29T11:18:49.425 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction test succeeded. 2024-03-29T11:18:49.425 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction 2024-03-29T11:18:49.577 INFO:teuthology.orchestra.run.smithi084.stdout: Preparing : 1/1 2024-03-29T11:18:49.682 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:18:50.100 INFO:teuthology.orchestra.run.smithi084.stdout: Running scriptlet: rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:18:50.374 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:18:50.374 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-03-29T11:18:50.374 INFO:teuthology.orchestra.run.smithi084.stdout:Installed: 2024-03-29T11:18:50.374 INFO:teuthology.orchestra.run.smithi084.stdout: rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:18:50.375 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-03-29T11:18:50.375 INFO:teuthology.orchestra.run.smithi084.stdout:Complete! 2024-03-29T11:18:50.609 DEBUG:teuthology.parallel:result is None 2024-03-29T11:18:50.609 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=5dcb190da270440ef141d088eb27a9fdb8afc5f2 2024-03-29T11:18:50.749 DEBUG:teuthology.orchestra.run.smithi084:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-29T11:18:50.769 INFO:teuthology.orchestra.run.smithi084.stdout:19.0.0-1439.g5dcb190d.el9 2024-03-29T11:18:50.770 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-1439.g5dcb190d.el9 2024-03-29T11:18:50.770 INFO:teuthology.task.install:The correct ceph version 19.0.0-1439.g5dcb190d is installed. 2024-03-29T11:18:50.772 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=5dcb190da270440ef141d088eb27a9fdb8afc5f2 2024-03-29T11:18:50.984 DEBUG:teuthology.orchestra.run.smithi179:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-29T11:18:51.009 INFO:teuthology.orchestra.run.smithi179.stdout:19.0.0-1439.g5dcb190d.el9 2024-03-29T11:18:51.009 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-1439.g5dcb190d.el9 2024-03-29T11:18:51.010 INFO:teuthology.task.install:The correct ceph version 19.0.0-1439.g5dcb190d is installed. 2024-03-29T11:18:51.011 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-03-29T11:18:51.012 DEBUG:teuthology.orchestra.run.smithi084:> set -ex 2024-03-29T11:18:51.012 DEBUG:teuthology.orchestra.run.smithi084:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-29T11:18:51.040 DEBUG:teuthology.orchestra.run.smithi179:> set -ex 2024-03-29T11:18:51.040 DEBUG:teuthology.orchestra.run.smithi179:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-29T11:18:51.082 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-03-29T11:18:51.083 DEBUG:teuthology.orchestra.run.smithi084:> set -ex 2024-03-29T11:18:51.083 DEBUG:teuthology.orchestra.run.smithi084:> sudo dd of=/usr/bin/daemon-helper 2024-03-29T11:18:51.110 DEBUG:teuthology.orchestra.run.smithi084:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-29T11:18:51.177 DEBUG:teuthology.orchestra.run.smithi179:> set -ex 2024-03-29T11:18:51.177 DEBUG:teuthology.orchestra.run.smithi179:> sudo dd of=/usr/bin/daemon-helper 2024-03-29T11:18:51.209 DEBUG:teuthology.orchestra.run.smithi179:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-29T11:18:51.281 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-03-29T11:18:51.282 DEBUG:teuthology.orchestra.run.smithi084:> set -ex 2024-03-29T11:18:51.282 DEBUG:teuthology.orchestra.run.smithi084:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-29T11:18:51.308 DEBUG:teuthology.orchestra.run.smithi084:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-29T11:18:51.375 DEBUG:teuthology.orchestra.run.smithi179:> set -ex 2024-03-29T11:18:51.375 DEBUG:teuthology.orchestra.run.smithi179:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-29T11:18:51.407 DEBUG:teuthology.orchestra.run.smithi179:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-29T11:18:51.479 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-03-29T11:18:51.479 DEBUG:teuthology.orchestra.run.smithi084:> set -ex 2024-03-29T11:18:51.479 DEBUG:teuthology.orchestra.run.smithi084:> sudo dd of=/usr/bin/stdin-killer 2024-03-29T11:18:51.506 DEBUG:teuthology.orchestra.run.smithi084:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-29T11:18:51.573 DEBUG:teuthology.orchestra.run.smithi179:> set -ex 2024-03-29T11:18:51.573 DEBUG:teuthology.orchestra.run.smithi179:> sudo dd of=/usr/bin/stdin-killer 2024-03-29T11:18:51.604 DEBUG:teuthology.orchestra.run.smithi179:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-29T11:18:51.674 INFO:teuthology.run_tasks:Running task ceph... 2024-03-29T11:18:51.759 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2024-03-29T11:18:51.759 DEBUG:teuthology.orchestra.run.smithi084:> sudo chmod 777 /var/log/ceph 2024-03-29T11:18:51.762 DEBUG:teuthology.orchestra.run.smithi179:> sudo chmod 777 /var/log/ceph 2024-03-29T11:18:51.792 INFO:tasks.ceph:Disabling ceph logrotate... 2024-03-29T11:18:51.792 DEBUG:teuthology.orchestra.run.smithi084:> sudo rm -f -- /etc/logrotate.d/ceph 2024-03-29T11:18:51.829 DEBUG:teuthology.orchestra.run.smithi179:> sudo rm -f -- /etc/logrotate.d/ceph 2024-03-29T11:18:51.863 INFO:tasks.ceph:Creating extra log directories... 2024-03-29T11:18:51.864 DEBUG:teuthology.orchestra.run.smithi084:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-03-29T11:18:51.896 DEBUG:teuthology.orchestra.run.smithi179:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-03-29T11:18:51.943 INFO:tasks.ceph:Creating ceph cluster ceph... 2024-03-29T11:18:51.943 INFO:tasks.ceph:config {'conf': {'client': {'debug monc': 20, 'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'global': {'lockdep': True, 'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms bind msgr2': False, 'ms inject socket failures': 5000, 'ms type': 'async'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'mon warn on pool no app': False, 'paxos service trim min': 10}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'snappy', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug monc': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd class default list': '*', 'osd class load list': '*', 'osd client watch timeout': 120, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd shutdown pgref assert': True}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME', 'reached quota', 'overall HEALTH_', '\\(CACHE_POOL_NO_HIT_SET\\)', '\\(POOL_FULL\\)', '\\(SMALLER_PGP_NUM\\)', '\\(SLOW_OPS\\)', '\\(CACHE_POOL_NEAR_FULL\\)', '\\(POOL_APP_NOT_ENABLED\\)', '\\(PG_AVAILABILITY\\)', '\\(OBJECT_MISPLACED\\)', 'slow request'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': False, 'mon_bind_addrvec': True} 2024-03-29T11:18:51.943 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622699', 'branch': 'main', 'description': 'rados:verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-snappy rados tasks/rados_api_tests validater/lockdep}', 'email': None, 'first_in_suite': False, 'job_id': '7622699', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'openstack': [{'volumes': {'count': 4, 'size': 10}}, {'volumes': {'count': 4, 'size': 10}}], 'os_type': 'centos', 'os_version': '9.stream', 'overrides': {'admin_socket': {'branch': 'main'}, 'ceph': {'conf': {'client': {'debug monc': 20, 'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'global': {'lockdep': True, 'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms bind msgr2': False, 'ms inject socket failures': 5000, 'ms type': 'async'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'mon warn on pool no app': False, 'paxos service trim min': 10}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'snappy', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug monc': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd class default list': '*', 'osd class load list': '*', 'osd client watch timeout': 120, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd shutdown pgref assert': True}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME', 'reached quota', 'overall HEALTH_', '\\(CACHE_POOL_NO_HIT_SET\\)', '\\(POOL_FULL\\)', '\\(SMALLER_PGP_NUM\\)', '\\(SLOW_OPS\\)', '\\(CACHE_POOL_NEAR_FULL\\)', '\\(POOL_APP_NOT_ENABLED\\)', '\\(PG_AVAILABILITY\\)', '\\(OBJECT_MISPLACED\\)', 'slow request'], 'mon_bind_msgr2': False, 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}}}, 'install': {'ceph': {'flavor': 'default', 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2'}}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'main', 'sha1': '1fb5d10608dccee73edd6ef8a2c7b804a097c324'}}, 'owner': 'scheduled_amathuri@teuthology', 'priority': 150, 'repo': 'https://git.ceph.com/ceph-ci.git', 'roles': [['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'], ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b']], 'seed': 8152, 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2', 'sleep_before_teardown': 0, 'subset': '99/1000', 'suite': 'rados:verify', 'suite_branch': 'main', 'suite_path': '/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://git.ceph.com/ceph-ci.git', 'suite_sha1': '1fb5d10608dccee73edd6ef8a2c7b804a097c324', 'targets': {'smithi084.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBEHXNFai/jEmct9ZIBZFZkJRhsklU2T6DhlQYD7OOFnU/7MBtkG1eum5vdqLyZuTMzqVORCX+BgUC7grhC+ksv4=', 'smithi179.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPL2qn7wt1ESWLkdih7XgUUtWDFzamLA5GFZpkPo13TiiE7jP24czk2rm6vPwLHGZeXugXdNWG/bPvTWDQXA12I='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': None}, {'workunit': {'clients': {'client.0': ['rados/test.sh']}, 'env': {'ALLOW_TIMEOUTS': '1'}, 'timeout': '6h'}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': 'e691533f9cbb33d85b2187bba20d7102f098636d', 'timestamp': '2024-03-26_05:45:44', 'tube': 'smithi', 'user': 'amathuri', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885'} 2024-03-29T11:18:51.944 DEBUG:teuthology.orchestra.run.smithi084:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-03-29T11:18:51.963 DEBUG:teuthology.orchestra.run.smithi179:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-03-29T11:18:51.990 DEBUG:teuthology.orchestra.run.smithi084:> sudo install -d -m0777 -- /var/run/ceph 2024-03-29T11:18:52.018 DEBUG:teuthology.orchestra.run.smithi179:> sudo install -d -m0777 -- /var/run/ceph 2024-03-29T11:18:52.058 DEBUG:teuthology.orchestra.run.smithi084:> set -ex 2024-03-29T11:18:52.059 DEBUG:teuthology.orchestra.run.smithi084:> dd if=/scratch_devs of=/dev/stdout 2024-03-29T11:18:52.098 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-29T11:18:52.098 DEBUG:teuthology.orchestra.run.smithi084:> stat /dev/vg_nvme/lv_1 2024-03-29T11:18:52.153 INFO:teuthology.orchestra.run.smithi084.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-29T11:18:52.153 INFO:teuthology.orchestra.run.smithi084.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-29T11:18:52.153 INFO:teuthology.orchestra.run.smithi084.stdout:Device: 5h/5d Inode: 668 Links: 1 2024-03-29T11:18:52.153 INFO:teuthology.orchestra.run.smithi084.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-29T11:18:52.154 INFO:teuthology.orchestra.run.smithi084.stdout:Context: system_u:object_r:device_t:s0 2024-03-29T11:18:52.154 INFO:teuthology.orchestra.run.smithi084.stdout:Access: 2024-03-29 11:18:46.855156459 +0000 2024-03-29T11:18:52.154 INFO:teuthology.orchestra.run.smithi084.stdout:Modify: 2024-03-29 11:15:25.034917656 +0000 2024-03-29T11:18:52.154 INFO:teuthology.orchestra.run.smithi084.stdout:Change: 2024-03-29 11:15:25.034917656 +0000 2024-03-29T11:18:52.154 INFO:teuthology.orchestra.run.smithi084.stdout: Birth: 2024-03-29 11:15:25.034917656 +0000 2024-03-29T11:18:52.154 DEBUG:teuthology.orchestra.run.smithi084:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-29T11:18:52.219 INFO:teuthology.orchestra.run.smithi084.stderr:1+0 records in 2024-03-29T11:18:52.219 INFO:teuthology.orchestra.run.smithi084.stderr:1+0 records out 2024-03-29T11:18:52.219 INFO:teuthology.orchestra.run.smithi084.stderr:512 bytes copied, 0.000215933 s, 2.4 MB/s 2024-03-29T11:18:52.220 DEBUG:teuthology.orchestra.run.smithi084:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-29T11:18:52.276 DEBUG:teuthology.orchestra.run.smithi084:> stat /dev/vg_nvme/lv_2 2024-03-29T11:18:52.331 INFO:teuthology.orchestra.run.smithi084.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-29T11:18:52.331 INFO:teuthology.orchestra.run.smithi084.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-29T11:18:52.331 INFO:teuthology.orchestra.run.smithi084.stdout:Device: 5h/5d Inode: 684 Links: 1 2024-03-29T11:18:52.332 INFO:teuthology.orchestra.run.smithi084.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-29T11:18:52.332 INFO:teuthology.orchestra.run.smithi084.stdout:Context: system_u:object_r:device_t:s0 2024-03-29T11:18:52.332 INFO:teuthology.orchestra.run.smithi084.stdout:Access: 2024-03-29 11:18:46.855156459 +0000 2024-03-29T11:18:52.332 INFO:teuthology.orchestra.run.smithi084.stdout:Modify: 2024-03-29 11:15:25.370911486 +0000 2024-03-29T11:18:52.332 INFO:teuthology.orchestra.run.smithi084.stdout:Change: 2024-03-29 11:15:25.370911486 +0000 2024-03-29T11:18:52.332 INFO:teuthology.orchestra.run.smithi084.stdout: Birth: 2024-03-29 11:15:25.370911486 +0000 2024-03-29T11:18:52.332 DEBUG:teuthology.orchestra.run.smithi084:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-29T11:18:52.397 INFO:teuthology.orchestra.run.smithi084.stderr:1+0 records in 2024-03-29T11:18:52.397 INFO:teuthology.orchestra.run.smithi084.stderr:1+0 records out 2024-03-29T11:18:52.397 INFO:teuthology.orchestra.run.smithi084.stderr:512 bytes copied, 0.000209639 s, 2.4 MB/s 2024-03-29T11:18:52.398 DEBUG:teuthology.orchestra.run.smithi084:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-29T11:18:52.454 DEBUG:teuthology.orchestra.run.smithi084:> stat /dev/vg_nvme/lv_3 2024-03-29T11:18:52.509 INFO:teuthology.orchestra.run.smithi084.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-29T11:18:52.509 INFO:teuthology.orchestra.run.smithi084.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-29T11:18:52.509 INFO:teuthology.orchestra.run.smithi084.stdout:Device: 5h/5d Inode: 704 Links: 1 2024-03-29T11:18:52.509 INFO:teuthology.orchestra.run.smithi084.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-29T11:18:52.509 INFO:teuthology.orchestra.run.smithi084.stdout:Context: system_u:object_r:device_t:s0 2024-03-29T11:18:52.509 INFO:teuthology.orchestra.run.smithi084.stdout:Access: 2024-03-29 11:18:46.856156441 +0000 2024-03-29T11:18:52.510 INFO:teuthology.orchestra.run.smithi084.stdout:Modify: 2024-03-29 11:15:25.721905042 +0000 2024-03-29T11:18:52.510 INFO:teuthology.orchestra.run.smithi084.stdout:Change: 2024-03-29 11:15:25.721905042 +0000 2024-03-29T11:18:52.510 INFO:teuthology.orchestra.run.smithi084.stdout: Birth: 2024-03-29 11:15:25.721905042 +0000 2024-03-29T11:18:52.510 DEBUG:teuthology.orchestra.run.smithi084:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-29T11:18:52.574 INFO:teuthology.orchestra.run.smithi084.stderr:1+0 records in 2024-03-29T11:18:52.575 INFO:teuthology.orchestra.run.smithi084.stderr:1+0 records out 2024-03-29T11:18:52.575 INFO:teuthology.orchestra.run.smithi084.stderr:512 bytes copied, 0.000241142 s, 2.1 MB/s 2024-03-29T11:18:52.576 DEBUG:teuthology.orchestra.run.smithi084:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-29T11:18:52.632 DEBUG:teuthology.orchestra.run.smithi084:> stat /dev/vg_nvme/lv_4 2024-03-29T11:18:52.687 INFO:teuthology.orchestra.run.smithi084.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-29T11:18:52.687 INFO:teuthology.orchestra.run.smithi084.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-29T11:18:52.687 INFO:teuthology.orchestra.run.smithi084.stdout:Device: 5h/5d Inode: 722 Links: 1 2024-03-29T11:18:52.687 INFO:teuthology.orchestra.run.smithi084.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-29T11:18:52.688 INFO:teuthology.orchestra.run.smithi084.stdout:Context: system_u:object_r:device_t:s0 2024-03-29T11:18:52.688 INFO:teuthology.orchestra.run.smithi084.stdout:Access: 2024-03-29 11:18:46.856156441 +0000 2024-03-29T11:18:52.688 INFO:teuthology.orchestra.run.smithi084.stdout:Modify: 2024-03-29 11:15:26.244895438 +0000 2024-03-29T11:18:52.688 INFO:teuthology.orchestra.run.smithi084.stdout:Change: 2024-03-29 11:15:26.244895438 +0000 2024-03-29T11:18:52.688 INFO:teuthology.orchestra.run.smithi084.stdout: Birth: 2024-03-29 11:15:26.244895438 +0000 2024-03-29T11:18:52.688 DEBUG:teuthology.orchestra.run.smithi084:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-29T11:18:52.753 INFO:teuthology.orchestra.run.smithi084.stderr:1+0 records in 2024-03-29T11:18:52.753 INFO:teuthology.orchestra.run.smithi084.stderr:1+0 records out 2024-03-29T11:18:52.753 INFO:teuthology.orchestra.run.smithi084.stderr:512 bytes copied, 0.000217236 s, 2.4 MB/s 2024-03-29T11:18:52.754 DEBUG:teuthology.orchestra.run.smithi084:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-29T11:18:52.810 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-29T11:18:52.810 DEBUG:teuthology.orchestra.run.smithi179:> set -ex 2024-03-29T11:18:52.810 DEBUG:teuthology.orchestra.run.smithi179:> dd if=/scratch_devs of=/dev/stdout 2024-03-29T11:18:52.828 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-29T11:18:52.829 DEBUG:teuthology.orchestra.run.smithi179:> stat /dev/vg_nvme/lv_1 2024-03-29T11:18:52.886 INFO:teuthology.orchestra.run.smithi179.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-29T11:18:52.886 INFO:teuthology.orchestra.run.smithi179.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-29T11:18:52.886 INFO:teuthology.orchestra.run.smithi179.stdout:Device: 5h/5d Inode: 665 Links: 1 2024-03-29T11:18:52.886 INFO:teuthology.orchestra.run.smithi179.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-29T11:18:52.886 INFO:teuthology.orchestra.run.smithi179.stdout:Context: system_u:object_r:device_t:s0 2024-03-29T11:18:52.886 INFO:teuthology.orchestra.run.smithi179.stdout:Access: 2024-03-29 11:18:40.400518829 +0000 2024-03-29T11:18:52.887 INFO:teuthology.orchestra.run.smithi179.stdout:Modify: 2024-03-29 11:15:26.740722995 +0000 2024-03-29T11:18:52.887 INFO:teuthology.orchestra.run.smithi179.stdout:Change: 2024-03-29 11:15:26.740722995 +0000 2024-03-29T11:18:52.887 INFO:teuthology.orchestra.run.smithi179.stdout: Birth: 2024-03-29 11:15:26.740722995 +0000 2024-03-29T11:18:52.887 DEBUG:teuthology.orchestra.run.smithi179:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-29T11:18:52.956 INFO:teuthology.orchestra.run.smithi179.stderr:1+0 records in 2024-03-29T11:18:52.956 INFO:teuthology.orchestra.run.smithi179.stderr:1+0 records out 2024-03-29T11:18:52.956 INFO:teuthology.orchestra.run.smithi179.stderr:512 bytes copied, 0.000224938 s, 2.3 MB/s 2024-03-29T11:18:52.957 DEBUG:teuthology.orchestra.run.smithi179:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-29T11:18:53.016 DEBUG:teuthology.orchestra.run.smithi179:> stat /dev/vg_nvme/lv_2 2024-03-29T11:18:53.073 INFO:teuthology.orchestra.run.smithi179.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-29T11:18:53.073 INFO:teuthology.orchestra.run.smithi179.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-29T11:18:53.073 INFO:teuthology.orchestra.run.smithi179.stdout:Device: 5h/5d Inode: 686 Links: 1 2024-03-29T11:18:53.073 INFO:teuthology.orchestra.run.smithi179.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-29T11:18:53.073 INFO:teuthology.orchestra.run.smithi179.stdout:Context: system_u:object_r:device_t:s0 2024-03-29T11:18:53.073 INFO:teuthology.orchestra.run.smithi179.stdout:Access: 2024-03-29 11:18:40.400518829 +0000 2024-03-29T11:18:53.073 INFO:teuthology.orchestra.run.smithi179.stdout:Modify: 2024-03-29 11:15:27.087715384 +0000 2024-03-29T11:18:53.073 INFO:teuthology.orchestra.run.smithi179.stdout:Change: 2024-03-29 11:15:27.087715384 +0000 2024-03-29T11:18:53.073 INFO:teuthology.orchestra.run.smithi179.stdout: Birth: 2024-03-29 11:15:27.087715384 +0000 2024-03-29T11:18:53.074 DEBUG:teuthology.orchestra.run.smithi179:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-29T11:18:53.143 INFO:teuthology.orchestra.run.smithi179.stderr:1+0 records in 2024-03-29T11:18:53.143 INFO:teuthology.orchestra.run.smithi179.stderr:1+0 records out 2024-03-29T11:18:53.143 INFO:teuthology.orchestra.run.smithi179.stderr:512 bytes copied, 0.000249587 s, 2.1 MB/s 2024-03-29T11:18:53.145 DEBUG:teuthology.orchestra.run.smithi179:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-29T11:18:53.203 DEBUG:teuthology.orchestra.run.smithi179:> stat /dev/vg_nvme/lv_3 2024-03-29T11:18:53.259 INFO:teuthology.orchestra.run.smithi179.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-29T11:18:53.259 INFO:teuthology.orchestra.run.smithi179.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-29T11:18:53.259 INFO:teuthology.orchestra.run.smithi179.stdout:Device: 5h/5d Inode: 703 Links: 1 2024-03-29T11:18:53.259 INFO:teuthology.orchestra.run.smithi179.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-29T11:18:53.259 INFO:teuthology.orchestra.run.smithi179.stdout:Context: system_u:object_r:device_t:s0 2024-03-29T11:18:53.259 INFO:teuthology.orchestra.run.smithi179.stdout:Access: 2024-03-29 11:18:40.401518807 +0000 2024-03-29T11:18:53.260 INFO:teuthology.orchestra.run.smithi179.stdout:Modify: 2024-03-29 11:15:27.430707860 +0000 2024-03-29T11:18:53.260 INFO:teuthology.orchestra.run.smithi179.stdout:Change: 2024-03-29 11:15:27.430707860 +0000 2024-03-29T11:18:53.260 INFO:teuthology.orchestra.run.smithi179.stdout: Birth: 2024-03-29 11:15:27.430707860 +0000 2024-03-29T11:18:53.260 DEBUG:teuthology.orchestra.run.smithi179:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-29T11:18:53.328 INFO:teuthology.orchestra.run.smithi179.stderr:1+0 records in 2024-03-29T11:18:53.328 INFO:teuthology.orchestra.run.smithi179.stderr:1+0 records out 2024-03-29T11:18:53.328 INFO:teuthology.orchestra.run.smithi179.stderr:512 bytes copied, 0.000198943 s, 2.6 MB/s 2024-03-29T11:18:53.329 DEBUG:teuthology.orchestra.run.smithi179:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-29T11:18:53.385 DEBUG:teuthology.orchestra.run.smithi179:> stat /dev/vg_nvme/lv_4 2024-03-29T11:18:53.440 INFO:teuthology.orchestra.run.smithi179.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-29T11:18:53.440 INFO:teuthology.orchestra.run.smithi179.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-29T11:18:53.440 INFO:teuthology.orchestra.run.smithi179.stdout:Device: 5h/5d Inode: 723 Links: 1 2024-03-29T11:18:53.441 INFO:teuthology.orchestra.run.smithi179.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-29T11:18:53.441 INFO:teuthology.orchestra.run.smithi179.stdout:Context: system_u:object_r:device_t:s0 2024-03-29T11:18:53.441 INFO:teuthology.orchestra.run.smithi179.stdout:Access: 2024-03-29 11:18:40.401518807 +0000 2024-03-29T11:18:53.441 INFO:teuthology.orchestra.run.smithi179.stdout:Modify: 2024-03-29 11:15:27.828699131 +0000 2024-03-29T11:18:53.441 INFO:teuthology.orchestra.run.smithi179.stdout:Change: 2024-03-29 11:15:27.828699131 +0000 2024-03-29T11:18:53.441 INFO:teuthology.orchestra.run.smithi179.stdout: Birth: 2024-03-29 11:15:27.828699131 +0000 2024-03-29T11:18:53.441 DEBUG:teuthology.orchestra.run.smithi179:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-29T11:18:53.506 INFO:teuthology.orchestra.run.smithi179.stderr:1+0 records in 2024-03-29T11:18:53.506 INFO:teuthology.orchestra.run.smithi179.stderr:1+0 records out 2024-03-29T11:18:53.507 INFO:teuthology.orchestra.run.smithi179.stderr:512 bytes copied, 0.000192715 s, 2.7 MB/s 2024-03-29T11:18:53.508 DEBUG:teuthology.orchestra.run.smithi179:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-29T11:18:53.563 INFO:tasks.ceph:osd dev map: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-29T11:18:53.563 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi084.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi179.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-03-29T11:18:53.563 INFO:tasks.ceph:Generating config... 2024-03-29T11:18:53.564 INFO:tasks.ceph:[client] debug monc = 20 2024-03-29T11:18:53.564 INFO:tasks.ceph:[client] debug ms = 1 2024-03-29T11:18:53.564 INFO:tasks.ceph:[client] debug objecter = 20 2024-03-29T11:18:53.565 INFO:tasks.ceph:[client] debug rados = 20 2024-03-29T11:18:53.565 INFO:tasks.ceph:[global] lockdep = True 2024-03-29T11:18:53.565 INFO:tasks.ceph:[global] mon client directed command retry = 5 2024-03-29T11:18:53.565 INFO:tasks.ceph:[global] mon election default strategy = 1 2024-03-29T11:18:53.565 INFO:tasks.ceph:[global] ms bind msgr2 = False 2024-03-29T11:18:53.565 INFO:tasks.ceph:[global] ms inject socket failures = 5000 2024-03-29T11:18:53.565 INFO:tasks.ceph:[global] ms type = async 2024-03-29T11:18:53.565 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-03-29T11:18:53.565 INFO:tasks.ceph:[mgr] debug ms = 1 2024-03-29T11:18:53.565 INFO:tasks.ceph:[mon] debug mon = 20 2024-03-29T11:18:53.565 INFO:tasks.ceph:[mon] debug ms = 1 2024-03-29T11:18:53.565 INFO:tasks.ceph:[mon] debug paxos = 20 2024-03-29T11:18:53.565 INFO:tasks.ceph:[mon] mon min osdmap epochs = 50 2024-03-29T11:18:53.565 INFO:tasks.ceph:[mon] mon osdmap full prune interval = 2 2024-03-29T11:18:53.565 INFO:tasks.ceph:[mon] mon osdmap full prune min = 15 2024-03-29T11:18:53.566 INFO:tasks.ceph:[mon] mon osdmap full prune txsize = 2 2024-03-29T11:18:53.566 INFO:tasks.ceph:[mon] mon scrub interval = 300 2024-03-29T11:18:53.566 INFO:tasks.ceph:[mon] mon warn on pool no app = False 2024-03-29T11:18:53.566 INFO:tasks.ceph:[mon] paxos service trim min = 10 2024-03-29T11:18:53.566 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2024-03-29T11:18:53.566 INFO:tasks.ceph:[osd] bluestore compression algorithm = snappy 2024-03-29T11:18:53.566 INFO:tasks.ceph:[osd] bluestore compression mode = aggressive 2024-03-29T11:18:53.566 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2024-03-29T11:18:53.566 INFO:tasks.ceph:[osd] bluestore zero block detection = True 2024-03-29T11:18:53.566 INFO:tasks.ceph:[osd] debug bluefs = 20 2024-03-29T11:18:53.566 INFO:tasks.ceph:[osd] debug bluestore = 20 2024-03-29T11:18:53.566 INFO:tasks.ceph:[osd] debug monc = 20 2024-03-29T11:18:53.566 INFO:tasks.ceph:[osd] debug ms = 1 2024-03-29T11:18:53.566 INFO:tasks.ceph:[osd] debug osd = 20 2024-03-29T11:18:53.567 INFO:tasks.ceph:[osd] debug rocksdb = 10 2024-03-29T11:18:53.567 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2024-03-29T11:18:53.567 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2024-03-29T11:18:53.567 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2024-03-29T11:18:53.567 INFO:tasks.ceph:[osd] osd class default list = * 2024-03-29T11:18:53.567 INFO:tasks.ceph:[osd] osd class load list = * 2024-03-29T11:18:53.567 INFO:tasks.ceph:[osd] osd client watch timeout = 120 2024-03-29T11:18:53.567 INFO:tasks.ceph:[osd] osd debug verify cached snaps = True 2024-03-29T11:18:53.567 INFO:tasks.ceph:[osd] osd debug verify missing on start = True 2024-03-29T11:18:53.567 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2024-03-29T11:18:53.567 INFO:tasks.ceph:[osd] osd mclock override recovery settings = True 2024-03-29T11:18:53.567 INFO:tasks.ceph:[osd] osd mclock profile = high_recovery_ops 2024-03-29T11:18:53.567 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2024-03-29T11:18:53.567 INFO:tasks.ceph:[osd] osd op queue = debug_random 2024-03-29T11:18:53.567 INFO:tasks.ceph:[osd] osd op queue cut off = debug_random 2024-03-29T11:18:53.568 INFO:tasks.ceph:[osd] osd shutdown pgref assert = True 2024-03-29T11:18:53.568 INFO:tasks.ceph:Setting up mon.a... 2024-03-29T11:18:53.568 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2024-03-29T11:18:53.622 INFO:teuthology.orchestra.run.smithi084.stdout:creating /etc/ceph/ceph.keyring 2024-03-29T11:18:53.625 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2024-03-29T11:18:53.670 DEBUG:teuthology.orchestra.run.smithi084:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-03-29T11:18:53.698 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '[v1:172.21.15.84:6789]'), ('mon.c', '[v1:172.21.15.84:6790]'), ('mon.b', '[v1:172.21.15.179:6789]')] 2024-03-29T11:18:53.698 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '[v1:172.21.15.84:6789],[v1:172.21.15.84:6790],[v1:172.21.15.179:6789]', 'lockdep': True, 'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms bind msgr2': False, 'ms inject socket failures': 5000, 'ms type': 'async'}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bluestore block size': 96636764160, 'bluestore compression algorithm': 'snappy', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug monc': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd class default list': '*', 'osd class load list': '*', 'osd client watch timeout': 120, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd mclock override recovery settings': True, 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd shutdown pgref assert': True}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'mon warn on pool no app': False, 'paxos service trim min': 10}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'debug monc': 20, 'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'mon.a': {}, 'mon.c': {}, 'mon.b': {}} 2024-03-29T11:18:53.699 DEBUG:teuthology.orchestra.run.smithi084:> set -ex 2024-03-29T11:18:53.699 DEBUG:teuthology.orchestra.run.smithi084:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-03-29T11:18:53.754 DEBUG:teuthology.orchestra.run.smithi084:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --addv a '[v1:172.21.15.84:6789]' --addv c '[v1:172.21.15.84:6790]' --addv b '[v1:172.21.15.179:6789]' --print /home/ubuntu/cephtest/ceph.monmap 2024-03-29T11:18:53.839 INFO:teuthology.orchestra.run.smithi084.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2024-03-29T11:18:53.840 INFO:teuthology.orchestra.run.smithi084.stdout:monmaptool: generated fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:18:53.840 INFO:teuthology.orchestra.run.smithi084.stdout:setting min_mon_release = quincy 2024-03-29T11:18:53.840 INFO:teuthology.orchestra.run.smithi084.stdout:epoch 0 2024-03-29T11:18:53.840 INFO:teuthology.orchestra.run.smithi084.stdout:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:18:53.840 INFO:teuthology.orchestra.run.smithi084.stdout:last_changed 2024-03-29T11:18:53.864025+0000 2024-03-29T11:18:53.840 INFO:teuthology.orchestra.run.smithi084.stdout:created 2024-03-29T11:18:53.864025+0000 2024-03-29T11:18:53.840 INFO:teuthology.orchestra.run.smithi084.stdout:min_mon_release 17 (quincy) 2024-03-29T11:18:53.840 INFO:teuthology.orchestra.run.smithi084.stdout:election_strategy: 1 2024-03-29T11:18:53.840 INFO:teuthology.orchestra.run.smithi084.stdout:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T11:18:53.840 INFO:teuthology.orchestra.run.smithi084.stdout:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T11:18:53.840 INFO:teuthology.orchestra.run.smithi084.stdout:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T11:18:53.840 INFO:teuthology.orchestra.run.smithi084.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (3 monitors) 2024-03-29T11:18:53.842 DEBUG:teuthology.orchestra.run.smithi084:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-03-29T11:18:53.856 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID 2c3e002f-9eb0-4860-853b-4ef1ef562213... 2024-03-29T11:18:53.857 DEBUG:teuthology.orchestra.run.smithi084:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-03-29T11:18:53.898 DEBUG:teuthology.orchestra.run.smithi179:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-03-29T11:18:53.945 INFO:teuthology.orchestra.run.smithi084.stdout:[global] 2024-03-29T11:18:53.946 INFO:teuthology.orchestra.run.smithi084.stdout: chdir = "" 2024-03-29T11:18:53.946 INFO:teuthology.orchestra.run.smithi084.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-03-29T11:18:53.946 INFO:teuthology.orchestra.run.smithi084.stdout: auth supported = cephx 2024-03-29T11:18:53.946 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-03-29T11:18:53.946 INFO:teuthology.orchestra.run.smithi084.stdout: filestore xattr use omap = true 2024-03-29T11:18:53.946 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-03-29T11:18:53.946 INFO:teuthology.orchestra.run.smithi084.stdout: mon clock drift allowed = 1.000 2024-03-29T11:18:53.946 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-03-29T11:18:53.946 INFO:teuthology.orchestra.run.smithi084.stdout: osd crush chooseleaf type = 0 2024-03-29T11:18:53.946 INFO:teuthology.orchestra.run.smithi084.stdout: auth debug = true 2024-03-29T11:18:53.946 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-03-29T11:18:53.946 INFO:teuthology.orchestra.run.smithi084.stdout: ms die on old message = true 2024-03-29T11:18:53.947 INFO:teuthology.orchestra.run.smithi084.stdout: ms die on bug = true 2024-03-29T11:18:53.947 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-03-29T11:18:53.947 INFO:teuthology.orchestra.run.smithi084.stdout: mon max pg per osd = 10000 # >= luminous 2024-03-29T11:18:53.947 INFO:teuthology.orchestra.run.smithi084.stdout: mon pg warn max object skew = 0 2024-03-29T11:18:53.947 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-03-29T11:18:53.947 INFO:teuthology.orchestra.run.smithi084.stdout: # disable pg_autoscaler by default for new pools 2024-03-29T11:18:53.947 INFO:teuthology.orchestra.run.smithi084.stdout: osd_pool_default_pg_autoscale_mode = off 2024-03-29T11:18:53.947 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-03-29T11:18:53.947 INFO:teuthology.orchestra.run.smithi084.stdout: osd pool default size = 2 2024-03-29T11:18:53.947 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-03-29T11:18:53.947 INFO:teuthology.orchestra.run.smithi084.stdout: mon osd allow primary affinity = true 2024-03-29T11:18:53.947 INFO:teuthology.orchestra.run.smithi084.stdout: mon osd allow pg remap = true 2024-03-29T11:18:53.947 INFO:teuthology.orchestra.run.smithi084.stdout: mon warn on legacy crush tunables = false 2024-03-29T11:18:53.947 INFO:teuthology.orchestra.run.smithi084.stdout: mon warn on crush straw calc version zero = false 2024-03-29T11:18:53.948 INFO:teuthology.orchestra.run.smithi084.stdout: mon warn on no sortbitwise = false 2024-03-29T11:18:53.948 INFO:teuthology.orchestra.run.smithi084.stdout: mon warn on osd down out interval zero = false 2024-03-29T11:18:53.948 INFO:teuthology.orchestra.run.smithi084.stdout: mon warn on too few osds = false 2024-03-29T11:18:53.948 INFO:teuthology.orchestra.run.smithi084.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-03-29T11:18:53.948 INFO:teuthology.orchestra.run.smithi084.stdout: mon_warn_on_pool_no_redundancy = false 2024-03-29T11:18:53.948 INFO:teuthology.orchestra.run.smithi084.stdout: mon_allow_pool_size_one = true 2024-03-29T11:18:53.948 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-03-29T11:18:53.948 INFO:teuthology.orchestra.run.smithi084.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-03-29T11:18:53.948 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-03-29T11:18:53.948 INFO:teuthology.orchestra.run.smithi084.stdout: osd default data pool replay window = 5 2024-03-29T11:18:53.948 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-03-29T11:18:53.948 INFO:teuthology.orchestra.run.smithi084.stdout: mon allow pool delete = true 2024-03-29T11:18:53.948 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-03-29T11:18:53.948 INFO:teuthology.orchestra.run.smithi084.stdout: mon cluster log file level = debug 2024-03-29T11:18:53.949 INFO:teuthology.orchestra.run.smithi084.stdout: debug asserts on shutdown = true 2024-03-29T11:18:53.949 INFO:teuthology.orchestra.run.smithi084.stdout: mon health detail to clog = false 2024-03-29T11:18:53.949 INFO:teuthology.orchestra.run.smithi084.stdout: mon host = "[v1:172.21.15.84:6789],[v1:172.21.15.84:6790],[v1:172.21.15.179:6789]" 2024-03-29T11:18:53.949 INFO:teuthology.orchestra.run.smithi084.stdout: lockdep = True 2024-03-29T11:18:53.949 INFO:teuthology.orchestra.run.smithi084.stdout: mon client directed command retry = 5 2024-03-29T11:18:53.949 INFO:teuthology.orchestra.run.smithi084.stdout: mon election default strategy = 1 2024-03-29T11:18:53.949 INFO:teuthology.orchestra.run.smithi084.stdout: ms bind msgr2 = False 2024-03-29T11:18:53.949 INFO:teuthology.orchestra.run.smithi084.stdout: ms inject socket failures = 5000 2024-03-29T11:18:53.949 INFO:teuthology.orchestra.run.smithi084.stdout: ms type = async 2024-03-29T11:18:53.949 INFO:teuthology.orchestra.run.smithi084.stdout: fsid = 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:18:53.949 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-03-29T11:18:53.949 INFO:teuthology.orchestra.run.smithi084.stdout:[osd] 2024-03-29T11:18:53.949 INFO:teuthology.orchestra.run.smithi084.stdout: osd journal size = 100 2024-03-29T11:18:53.950 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-03-29T11:18:53.950 INFO:teuthology.orchestra.run.smithi084.stdout: osd scrub load threshold = 5.0 2024-03-29T11:18:53.950 INFO:teuthology.orchestra.run.smithi084.stdout: osd scrub max interval = 600 2024-03-29T11:18:53.950 INFO:teuthology.orchestra.run.smithi084.stdout: osd mclock profile = high_recovery_ops 2024-03-29T11:18:53.950 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-03-29T11:18:53.950 INFO:teuthology.orchestra.run.smithi084.stdout: osd recover clone overlap = true 2024-03-29T11:18:53.950 INFO:teuthology.orchestra.run.smithi084.stdout: osd recovery max chunk = 1048576 2024-03-29T11:18:53.950 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-03-29T11:18:53.950 INFO:teuthology.orchestra.run.smithi084.stdout: osd debug shutdown = true 2024-03-29T11:18:53.950 INFO:teuthology.orchestra.run.smithi084.stdout: osd debug op order = true 2024-03-29T11:18:53.950 INFO:teuthology.orchestra.run.smithi084.stdout: osd debug verify stray on activate = true 2024-03-29T11:18:53.950 INFO:teuthology.orchestra.run.smithi084.stdout: osd debug trim objects = true 2024-03-29T11:18:53.950 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-03-29T11:18:53.950 INFO:teuthology.orchestra.run.smithi084.stdout: osd open classes on start = true 2024-03-29T11:18:53.951 INFO:teuthology.orchestra.run.smithi084.stdout: osd debug pg log writeout = true 2024-03-29T11:18:53.951 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-03-29T11:18:53.951 INFO:teuthology.orchestra.run.smithi084.stdout: osd deep scrub update digest min age = 30 2024-03-29T11:18:53.951 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-03-29T11:18:53.951 INFO:teuthology.orchestra.run.smithi084.stdout: osd map max advance = 10 2024-03-29T11:18:53.951 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-03-29T11:18:53.951 INFO:teuthology.orchestra.run.smithi084.stdout: journal zero on create = true 2024-03-29T11:18:53.951 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-03-29T11:18:53.951 INFO:teuthology.orchestra.run.smithi084.stdout: filestore ondisk finisher threads = 3 2024-03-29T11:18:53.951 INFO:teuthology.orchestra.run.smithi084.stdout: filestore apply finisher threads = 3 2024-03-29T11:18:53.951 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-03-29T11:18:53.951 INFO:teuthology.orchestra.run.smithi084.stdout: bdev debug aio = true 2024-03-29T11:18:53.951 INFO:teuthology.orchestra.run.smithi084.stdout: osd debug misdirected ops = true 2024-03-29T11:18:53.951 INFO:teuthology.orchestra.run.smithi084.stdout: bluestore block size = 96636764160 2024-03-29T11:18:53.951 INFO:teuthology.orchestra.run.smithi084.stdout: bluestore compression algorithm = snappy 2024-03-29T11:18:53.952 INFO:teuthology.orchestra.run.smithi084.stdout: bluestore compression mode = aggressive 2024-03-29T11:18:53.952 INFO:teuthology.orchestra.run.smithi084.stdout: bluestore fsck on mount = True 2024-03-29T11:18:53.952 INFO:teuthology.orchestra.run.smithi084.stdout: bluestore zero block detection = True 2024-03-29T11:18:53.952 INFO:teuthology.orchestra.run.smithi084.stdout: debug bluefs = 20 2024-03-29T11:18:53.952 INFO:teuthology.orchestra.run.smithi084.stdout: debug bluestore = 20 2024-03-29T11:18:53.952 INFO:teuthology.orchestra.run.smithi084.stdout: debug monc = 20 2024-03-29T11:18:53.952 INFO:teuthology.orchestra.run.smithi084.stdout: debug ms = 1 2024-03-29T11:18:53.952 INFO:teuthology.orchestra.run.smithi084.stdout: debug osd = 20 2024-03-29T11:18:53.952 INFO:teuthology.orchestra.run.smithi084.stdout: debug rocksdb = 10 2024-03-29T11:18:53.952 INFO:teuthology.orchestra.run.smithi084.stdout: mon osd backfillfull_ratio = 0.85 2024-03-29T11:18:53.952 INFO:teuthology.orchestra.run.smithi084.stdout: mon osd full ratio = 0.9 2024-03-29T11:18:53.952 INFO:teuthology.orchestra.run.smithi084.stdout: mon osd nearfull ratio = 0.8 2024-03-29T11:18:53.952 INFO:teuthology.orchestra.run.smithi084.stdout: osd class default list = * 2024-03-29T11:18:53.952 INFO:teuthology.orchestra.run.smithi084.stdout: osd class load list = * 2024-03-29T11:18:53.952 INFO:teuthology.orchestra.run.smithi084.stdout: osd client watch timeout = 120 2024-03-29T11:18:53.952 INFO:teuthology.orchestra.run.smithi084.stdout: osd debug verify cached snaps = True 2024-03-29T11:18:53.953 INFO:teuthology.orchestra.run.smithi084.stdout: osd debug verify missing on start = True 2024-03-29T11:18:53.953 INFO:teuthology.orchestra.run.smithi084.stdout: osd failsafe full ratio = 0.95 2024-03-29T11:18:53.953 INFO:teuthology.orchestra.run.smithi084.stdout: osd mclock override recovery settings = True 2024-03-29T11:18:53.953 INFO:teuthology.orchestra.run.smithi084.stdout: osd objectstore = bluestore 2024-03-29T11:18:53.953 INFO:teuthology.orchestra.run.smithi084.stdout: osd op queue = debug_random 2024-03-29T11:18:53.953 INFO:teuthology.orchestra.run.smithi084.stdout: osd op queue cut off = debug_random 2024-03-29T11:18:53.953 INFO:teuthology.orchestra.run.smithi084.stdout: osd shutdown pgref assert = True 2024-03-29T11:18:53.953 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-03-29T11:18:53.953 INFO:teuthology.orchestra.run.smithi084.stdout:[mgr] 2024-03-29T11:18:53.953 INFO:teuthology.orchestra.run.smithi084.stdout: debug ms = 1 2024-03-29T11:18:53.953 INFO:teuthology.orchestra.run.smithi084.stdout: debug mgr = 20 2024-03-29T11:18:53.953 INFO:teuthology.orchestra.run.smithi084.stdout: debug mon = 20 2024-03-29T11:18:53.953 INFO:teuthology.orchestra.run.smithi084.stdout: debug auth = 20 2024-03-29T11:18:53.953 INFO:teuthology.orchestra.run.smithi084.stdout: mon reweight min pgs per osd = 4 2024-03-29T11:18:53.953 INFO:teuthology.orchestra.run.smithi084.stdout: mon reweight min bytes per osd = 10 2024-03-29T11:18:53.954 INFO:teuthology.orchestra.run.smithi084.stdout: mgr/telemetry/nag = false 2024-03-29T11:18:53.954 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-03-29T11:18:53.954 INFO:teuthology.orchestra.run.smithi084.stdout:[mon] 2024-03-29T11:18:53.954 INFO:teuthology.orchestra.run.smithi084.stdout: debug ms = 1 2024-03-29T11:18:53.954 INFO:teuthology.orchestra.run.smithi084.stdout: debug mon = 20 2024-03-29T11:18:53.954 INFO:teuthology.orchestra.run.smithi084.stdout: debug paxos = 20 2024-03-29T11:18:53.954 INFO:teuthology.orchestra.run.smithi084.stdout: debug auth = 20 2024-03-29T11:18:53.954 INFO:teuthology.orchestra.run.smithi084.stdout: mon data avail warn = 5 2024-03-29T11:18:53.954 INFO:teuthology.orchestra.run.smithi084.stdout: mon mgr mkfs grace = 240 2024-03-29T11:18:53.954 INFO:teuthology.orchestra.run.smithi084.stdout: mon reweight min pgs per osd = 4 2024-03-29T11:18:53.954 INFO:teuthology.orchestra.run.smithi084.stdout: mon osd reporter subtree level = osd 2024-03-29T11:18:53.954 INFO:teuthology.orchestra.run.smithi084.stdout: mon osd prime pg temp = true 2024-03-29T11:18:53.954 INFO:teuthology.orchestra.run.smithi084.stdout: mon reweight min bytes per osd = 10 2024-03-29T11:18:53.955 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-03-29T11:18:53.955 INFO:teuthology.orchestra.run.smithi084.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-03-29T11:18:53.955 INFO:teuthology.orchestra.run.smithi084.stdout: auth mon ticket ttl = 660 # 11m 2024-03-29T11:18:53.955 INFO:teuthology.orchestra.run.smithi084.stdout: auth service ticket ttl = 240 # 4m 2024-03-29T11:18:53.955 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-03-29T11:18:53.955 INFO:teuthology.orchestra.run.smithi084.stdout: # don't complain about insecure global_id in the test suite 2024-03-29T11:18:53.955 INFO:teuthology.orchestra.run.smithi084.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-03-29T11:18:53.955 INFO:teuthology.orchestra.run.smithi084.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-03-29T11:18:53.955 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-03-29T11:18:53.955 INFO:teuthology.orchestra.run.smithi084.stdout: # 1m isn't quite enough 2024-03-29T11:18:53.955 INFO:teuthology.orchestra.run.smithi084.stdout: mon_down_mkfs_grace = 2m 2024-03-29T11:18:53.955 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-03-29T11:18:53.955 INFO:teuthology.orchestra.run.smithi084.stdout: mon_warn_on_filestore_osds = false 2024-03-29T11:18:53.955 INFO:teuthology.orchestra.run.smithi084.stdout: mon min osdmap epochs = 50 2024-03-29T11:18:53.955 INFO:teuthology.orchestra.run.smithi084.stdout: mon osdmap full prune interval = 2 2024-03-29T11:18:53.956 INFO:teuthology.orchestra.run.smithi084.stdout: mon osdmap full prune min = 15 2024-03-29T11:18:53.956 INFO:teuthology.orchestra.run.smithi084.stdout: mon osdmap full prune txsize = 2 2024-03-29T11:18:53.956 INFO:teuthology.orchestra.run.smithi084.stdout: mon scrub interval = 300 2024-03-29T11:18:53.956 INFO:teuthology.orchestra.run.smithi084.stdout: mon warn on pool no app = False 2024-03-29T11:18:53.956 INFO:teuthology.orchestra.run.smithi084.stdout: paxos service trim min = 10 2024-03-29T11:18:53.956 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-03-29T11:18:53.956 INFO:teuthology.orchestra.run.smithi084.stdout:[client] 2024-03-29T11:18:53.956 INFO:teuthology.orchestra.run.smithi084.stdout: rgw cache enabled = true 2024-03-29T11:18:53.956 INFO:teuthology.orchestra.run.smithi084.stdout: rgw enable ops log = true 2024-03-29T11:18:53.956 INFO:teuthology.orchestra.run.smithi084.stdout: rgw enable usage log = true 2024-03-29T11:18:53.956 INFO:teuthology.orchestra.run.smithi084.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-03-29T11:18:53.956 INFO:teuthology.orchestra.run.smithi084.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-03-29T11:18:53.956 INFO:teuthology.orchestra.run.smithi084.stdout: debug monc = 20 2024-03-29T11:18:53.956 INFO:teuthology.orchestra.run.smithi084.stdout: debug ms = 1 2024-03-29T11:18:53.957 INFO:teuthology.orchestra.run.smithi084.stdout: debug objecter = 20 2024-03-29T11:18:53.957 INFO:teuthology.orchestra.run.smithi084.stdout: debug rados = 20 2024-03-29T11:18:53.957 INFO:teuthology.orchestra.run.smithi084.stdout:[mon.a] 2024-03-29T11:18:53.957 INFO:teuthology.orchestra.run.smithi084.stdout:[mon.c] 2024-03-29T11:18:53.957 INFO:teuthology.orchestra.run.smithi084.stdout:[mon.b] 2024-03-29T11:18:53.957 INFO:teuthology.orchestra.run.smithi179.stdout:[global] 2024-03-29T11:18:53.957 INFO:teuthology.orchestra.run.smithi179.stdout: chdir = "" 2024-03-29T11:18:53.957 INFO:teuthology.orchestra.run.smithi179.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-03-29T11:18:53.957 INFO:teuthology.orchestra.run.smithi179.stdout: auth supported = cephx 2024-03-29T11:18:53.958 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-29T11:18:53.958 INFO:teuthology.orchestra.run.smithi179.stdout: filestore xattr use omap = true 2024-03-29T11:18:53.958 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-29T11:18:53.958 INFO:teuthology.orchestra.run.smithi179.stdout: mon clock drift allowed = 1.000 2024-03-29T11:18:53.958 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-29T11:18:53.958 INFO:teuthology.orchestra.run.smithi179.stdout: osd crush chooseleaf type = 0 2024-03-29T11:18:53.958 INFO:teuthology.orchestra.run.smithi179.stdout: auth debug = true 2024-03-29T11:18:53.958 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-29T11:18:53.958 INFO:teuthology.orchestra.run.smithi179.stdout: ms die on old message = true 2024-03-29T11:18:53.958 INFO:teuthology.orchestra.run.smithi179.stdout: ms die on bug = true 2024-03-29T11:18:53.958 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-29T11:18:53.958 INFO:teuthology.orchestra.run.smithi179.stdout: mon max pg per osd = 10000 # >= luminous 2024-03-29T11:18:53.958 INFO:teuthology.orchestra.run.smithi179.stdout: mon pg warn max object skew = 0 2024-03-29T11:18:53.958 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-29T11:18:53.958 INFO:teuthology.orchestra.run.smithi179.stdout: # disable pg_autoscaler by default for new pools 2024-03-29T11:18:53.959 INFO:teuthology.orchestra.run.smithi179.stdout: osd_pool_default_pg_autoscale_mode = off 2024-03-29T11:18:53.959 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-29T11:18:53.959 INFO:teuthology.orchestra.run.smithi179.stdout: osd pool default size = 2 2024-03-29T11:18:53.959 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-29T11:18:53.959 INFO:teuthology.orchestra.run.smithi179.stdout: mon osd allow primary affinity = true 2024-03-29T11:18:53.959 INFO:teuthology.orchestra.run.smithi179.stdout: mon osd allow pg remap = true 2024-03-29T11:18:53.959 INFO:teuthology.orchestra.run.smithi179.stdout: mon warn on legacy crush tunables = false 2024-03-29T11:18:53.959 INFO:teuthology.orchestra.run.smithi179.stdout: mon warn on crush straw calc version zero = false 2024-03-29T11:18:53.959 INFO:teuthology.orchestra.run.smithi179.stdout: mon warn on no sortbitwise = false 2024-03-29T11:18:53.959 INFO:teuthology.orchestra.run.smithi179.stdout: mon warn on osd down out interval zero = false 2024-03-29T11:18:53.959 INFO:teuthology.orchestra.run.smithi179.stdout: mon warn on too few osds = false 2024-03-29T11:18:53.959 INFO:teuthology.orchestra.run.smithi179.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-03-29T11:18:53.959 INFO:teuthology.orchestra.run.smithi179.stdout: mon_warn_on_pool_no_redundancy = false 2024-03-29T11:18:53.960 INFO:teuthology.orchestra.run.smithi179.stdout: mon_allow_pool_size_one = true 2024-03-29T11:18:53.960 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-29T11:18:53.960 INFO:teuthology.orchestra.run.smithi179.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-03-29T11:18:53.960 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-29T11:18:53.960 INFO:teuthology.orchestra.run.smithi179.stdout: osd default data pool replay window = 5 2024-03-29T11:18:53.960 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-29T11:18:53.960 INFO:teuthology.orchestra.run.smithi179.stdout: mon allow pool delete = true 2024-03-29T11:18:53.960 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-29T11:18:53.960 INFO:teuthology.orchestra.run.smithi179.stdout: mon cluster log file level = debug 2024-03-29T11:18:53.960 INFO:teuthology.orchestra.run.smithi179.stdout: debug asserts on shutdown = true 2024-03-29T11:18:53.960 INFO:teuthology.orchestra.run.smithi179.stdout: mon health detail to clog = false 2024-03-29T11:18:53.960 INFO:teuthology.orchestra.run.smithi179.stdout: mon host = "[v1:172.21.15.84:6789],[v1:172.21.15.84:6790],[v1:172.21.15.179:6789]" 2024-03-29T11:18:53.960 INFO:teuthology.orchestra.run.smithi179.stdout: lockdep = True 2024-03-29T11:18:53.960 INFO:teuthology.orchestra.run.smithi179.stdout: mon client directed command retry = 5 2024-03-29T11:18:53.960 INFO:teuthology.orchestra.run.smithi179.stdout: mon election default strategy = 1 2024-03-29T11:18:53.961 INFO:teuthology.orchestra.run.smithi179.stdout: ms bind msgr2 = False 2024-03-29T11:18:53.961 INFO:teuthology.orchestra.run.smithi179.stdout: ms inject socket failures = 5000 2024-03-29T11:18:53.961 INFO:teuthology.orchestra.run.smithi179.stdout: ms type = async 2024-03-29T11:18:53.961 INFO:teuthology.orchestra.run.smithi179.stdout: fsid = 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:18:53.961 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-29T11:18:53.961 INFO:teuthology.orchestra.run.smithi179.stdout:[osd] 2024-03-29T11:18:53.961 INFO:teuthology.orchestra.run.smithi179.stdout: osd journal size = 100 2024-03-29T11:18:53.961 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-29T11:18:53.961 INFO:teuthology.orchestra.run.smithi179.stdout: osd scrub load threshold = 5.0 2024-03-29T11:18:53.961 INFO:teuthology.orchestra.run.smithi179.stdout: osd scrub max interval = 600 2024-03-29T11:18:53.961 INFO:teuthology.orchestra.run.smithi179.stdout: osd mclock profile = high_recovery_ops 2024-03-29T11:18:53.961 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-29T11:18:53.961 INFO:teuthology.orchestra.run.smithi179.stdout: osd recover clone overlap = true 2024-03-29T11:18:53.961 INFO:teuthology.orchestra.run.smithi179.stdout: osd recovery max chunk = 1048576 2024-03-29T11:18:53.962 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-29T11:18:53.962 INFO:teuthology.orchestra.run.smithi179.stdout: osd debug shutdown = true 2024-03-29T11:18:53.962 INFO:teuthology.orchestra.run.smithi179.stdout: osd debug op order = true 2024-03-29T11:18:53.962 INFO:teuthology.orchestra.run.smithi179.stdout: osd debug verify stray on activate = true 2024-03-29T11:18:53.962 INFO:teuthology.orchestra.run.smithi179.stdout: osd debug trim objects = true 2024-03-29T11:18:53.962 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-29T11:18:53.962 INFO:teuthology.orchestra.run.smithi179.stdout: osd open classes on start = true 2024-03-29T11:18:53.962 INFO:teuthology.orchestra.run.smithi179.stdout: osd debug pg log writeout = true 2024-03-29T11:18:53.962 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-29T11:18:53.962 INFO:teuthology.orchestra.run.smithi179.stdout: osd deep scrub update digest min age = 30 2024-03-29T11:18:53.962 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-29T11:18:53.962 INFO:teuthology.orchestra.run.smithi179.stdout: osd map max advance = 10 2024-03-29T11:18:53.962 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-29T11:18:53.962 INFO:teuthology.orchestra.run.smithi179.stdout: journal zero on create = true 2024-03-29T11:18:53.963 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-29T11:18:53.963 INFO:teuthology.orchestra.run.smithi179.stdout: filestore ondisk finisher threads = 3 2024-03-29T11:18:53.963 INFO:teuthology.orchestra.run.smithi179.stdout: filestore apply finisher threads = 3 2024-03-29T11:18:53.963 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-29T11:18:53.963 INFO:teuthology.orchestra.run.smithi179.stdout: bdev debug aio = true 2024-03-29T11:18:53.963 INFO:teuthology.orchestra.run.smithi179.stdout: osd debug misdirected ops = true 2024-03-29T11:18:53.963 INFO:teuthology.orchestra.run.smithi179.stdout: bluestore block size = 96636764160 2024-03-29T11:18:53.963 INFO:teuthology.orchestra.run.smithi179.stdout: bluestore compression algorithm = snappy 2024-03-29T11:18:53.963 INFO:teuthology.orchestra.run.smithi179.stdout: bluestore compression mode = aggressive 2024-03-29T11:18:53.963 INFO:teuthology.orchestra.run.smithi179.stdout: bluestore fsck on mount = True 2024-03-29T11:18:53.963 INFO:teuthology.orchestra.run.smithi179.stdout: bluestore zero block detection = True 2024-03-29T11:18:53.963 INFO:teuthology.orchestra.run.smithi179.stdout: debug bluefs = 20 2024-03-29T11:18:53.964 INFO:teuthology.orchestra.run.smithi179.stdout: debug bluestore = 20 2024-03-29T11:18:53.964 INFO:teuthology.orchestra.run.smithi179.stdout: debug monc = 20 2024-03-29T11:18:53.964 INFO:teuthology.orchestra.run.smithi179.stdout: debug ms = 1 2024-03-29T11:18:53.964 INFO:teuthology.orchestra.run.smithi179.stdout: debug osd = 20 2024-03-29T11:18:53.964 INFO:teuthology.orchestra.run.smithi179.stdout: debug rocksdb = 10 2024-03-29T11:18:53.964 INFO:teuthology.orchestra.run.smithi179.stdout: mon osd backfillfull_ratio = 0.85 2024-03-29T11:18:53.964 INFO:teuthology.orchestra.run.smithi179.stdout: mon osd full ratio = 0.9 2024-03-29T11:18:53.964 INFO:teuthology.orchestra.run.smithi179.stdout: mon osd nearfull ratio = 0.8 2024-03-29T11:18:53.964 INFO:teuthology.orchestra.run.smithi179.stdout: osd class default list = * 2024-03-29T11:18:53.964 INFO:teuthology.orchestra.run.smithi179.stdout: osd class load list = * 2024-03-29T11:18:53.964 INFO:teuthology.orchestra.run.smithi179.stdout: osd client watch timeout = 120 2024-03-29T11:18:53.964 INFO:teuthology.orchestra.run.smithi179.stdout: osd debug verify cached snaps = True 2024-03-29T11:18:53.964 INFO:teuthology.orchestra.run.smithi179.stdout: osd debug verify missing on start = True 2024-03-29T11:18:53.964 INFO:teuthology.orchestra.run.smithi179.stdout: osd failsafe full ratio = 0.95 2024-03-29T11:18:53.964 INFO:teuthology.orchestra.run.smithi179.stdout: osd mclock override recovery settings = True 2024-03-29T11:18:53.965 INFO:teuthology.orchestra.run.smithi179.stdout: osd objectstore = bluestore 2024-03-29T11:18:53.965 INFO:teuthology.orchestra.run.smithi179.stdout: osd op queue = debug_random 2024-03-29T11:18:53.965 INFO:teuthology.orchestra.run.smithi179.stdout: osd op queue cut off = debug_random 2024-03-29T11:18:53.965 INFO:teuthology.orchestra.run.smithi179.stdout: osd shutdown pgref assert = True 2024-03-29T11:18:53.965 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-29T11:18:53.965 INFO:teuthology.orchestra.run.smithi179.stdout:[mgr] 2024-03-29T11:18:53.965 INFO:teuthology.orchestra.run.smithi179.stdout: debug ms = 1 2024-03-29T11:18:53.965 INFO:teuthology.orchestra.run.smithi179.stdout: debug mgr = 20 2024-03-29T11:18:53.965 INFO:teuthology.orchestra.run.smithi179.stdout: debug mon = 20 2024-03-29T11:18:53.965 INFO:teuthology.orchestra.run.smithi179.stdout: debug auth = 20 2024-03-29T11:18:53.965 INFO:teuthology.orchestra.run.smithi179.stdout: mon reweight min pgs per osd = 4 2024-03-29T11:18:53.965 INFO:teuthology.orchestra.run.smithi179.stdout: mon reweight min bytes per osd = 10 2024-03-29T11:18:53.965 INFO:teuthology.orchestra.run.smithi179.stdout: mgr/telemetry/nag = false 2024-03-29T11:18:53.965 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-29T11:18:53.966 INFO:teuthology.orchestra.run.smithi179.stdout:[mon] 2024-03-29T11:18:53.966 INFO:teuthology.orchestra.run.smithi179.stdout: debug ms = 1 2024-03-29T11:18:53.966 INFO:teuthology.orchestra.run.smithi179.stdout: debug mon = 20 2024-03-29T11:18:53.966 INFO:teuthology.orchestra.run.smithi179.stdout: debug paxos = 20 2024-03-29T11:18:53.966 INFO:teuthology.orchestra.run.smithi179.stdout: debug auth = 20 2024-03-29T11:18:53.966 INFO:teuthology.orchestra.run.smithi179.stdout: mon data avail warn = 5 2024-03-29T11:18:53.966 INFO:teuthology.orchestra.run.smithi179.stdout: mon mgr mkfs grace = 240 2024-03-29T11:18:53.966 INFO:teuthology.orchestra.run.smithi179.stdout: mon reweight min pgs per osd = 4 2024-03-29T11:18:53.966 INFO:teuthology.orchestra.run.smithi179.stdout: mon osd reporter subtree level = osd 2024-03-29T11:18:53.966 INFO:teuthology.orchestra.run.smithi179.stdout: mon osd prime pg temp = true 2024-03-29T11:18:53.966 INFO:teuthology.orchestra.run.smithi179.stdout: mon reweight min bytes per osd = 10 2024-03-29T11:18:53.966 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-29T11:18:53.966 INFO:teuthology.orchestra.run.smithi179.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-03-29T11:18:53.967 INFO:teuthology.orchestra.run.smithi179.stdout: auth mon ticket ttl = 660 # 11m 2024-03-29T11:18:53.967 INFO:teuthology.orchestra.run.smithi179.stdout: auth service ticket ttl = 240 # 4m 2024-03-29T11:18:53.967 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-29T11:18:53.967 INFO:teuthology.orchestra.run.smithi179.stdout: # don't complain about insecure global_id in the test suite 2024-03-29T11:18:53.967 INFO:teuthology.orchestra.run.smithi179.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-03-29T11:18:53.967 INFO:teuthology.orchestra.run.smithi179.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-03-29T11:18:53.967 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-29T11:18:53.967 INFO:teuthology.orchestra.run.smithi179.stdout: # 1m isn't quite enough 2024-03-29T11:18:53.967 INFO:teuthology.orchestra.run.smithi179.stdout: mon_down_mkfs_grace = 2m 2024-03-29T11:18:53.967 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-29T11:18:53.967 INFO:teuthology.orchestra.run.smithi179.stdout: mon_warn_on_filestore_osds = false 2024-03-29T11:18:53.967 INFO:teuthology.orchestra.run.smithi179.stdout: mon min osdmap epochs = 50 2024-03-29T11:18:53.967 INFO:teuthology.orchestra.run.smithi179.stdout: mon osdmap full prune interval = 2 2024-03-29T11:18:53.967 INFO:teuthology.orchestra.run.smithi179.stdout: mon osdmap full prune min = 15 2024-03-29T11:18:53.967 INFO:teuthology.orchestra.run.smithi179.stdout: mon osdmap full prune txsize = 2 2024-03-29T11:18:53.968 INFO:teuthology.orchestra.run.smithi179.stdout: mon scrub interval = 300 2024-03-29T11:18:53.968 INFO:teuthology.orchestra.run.smithi179.stdout: mon warn on pool no app = False 2024-03-29T11:18:53.968 INFO:teuthology.orchestra.run.smithi179.stdout: paxos service trim min = 10 2024-03-29T11:18:53.968 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-29T11:18:53.968 INFO:teuthology.orchestra.run.smithi179.stdout:[client] 2024-03-29T11:18:53.968 INFO:teuthology.orchestra.run.smithi179.stdout: rgw cache enabled = true 2024-03-29T11:18:53.968 INFO:teuthology.orchestra.run.smithi179.stdout: rgw enable ops log = true 2024-03-29T11:18:53.968 INFO:teuthology.orchestra.run.smithi179.stdout: rgw enable usage log = true 2024-03-29T11:18:53.968 INFO:teuthology.orchestra.run.smithi179.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-03-29T11:18:53.968 INFO:teuthology.orchestra.run.smithi179.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-03-29T11:18:53.968 INFO:teuthology.orchestra.run.smithi179.stdout: debug monc = 20 2024-03-29T11:18:53.968 INFO:teuthology.orchestra.run.smithi179.stdout: debug ms = 1 2024-03-29T11:18:53.968 INFO:teuthology.orchestra.run.smithi179.stdout: debug objecter = 20 2024-03-29T11:18:53.968 INFO:teuthology.orchestra.run.smithi179.stdout: debug rados = 20 2024-03-29T11:18:53.968 INFO:teuthology.orchestra.run.smithi179.stdout:[mon.a] 2024-03-29T11:18:53.969 INFO:teuthology.orchestra.run.smithi179.stdout:[mon.c] 2024-03-29T11:18:53.969 INFO:teuthology.orchestra.run.smithi179.stdout:[mon.b] 2024-03-29T11:18:53.970 INFO:tasks.ceph:Creating admin key on mon.a... 2024-03-29T11:18:53.970 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2024-03-29T11:18:54.066 INFO:tasks.ceph:Copying monmap to all nodes... 2024-03-29T11:18:54.066 DEBUG:teuthology.orchestra.run.smithi084:> set -ex 2024-03-29T11:18:54.067 DEBUG:teuthology.orchestra.run.smithi084:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2024-03-29T11:18:54.081 DEBUG:teuthology.orchestra.run.smithi084:> set -ex 2024-03-29T11:18:54.081 DEBUG:teuthology.orchestra.run.smithi084:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2024-03-29T11:18:54.136 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi084.front.sepia.ceph.com 2024-03-29T11:18:54.136 DEBUG:teuthology.orchestra.run.smithi084:> set -ex 2024-03-29T11:18:54.136 DEBUG:teuthology.orchestra.run.smithi084:> sudo dd of=/etc/ceph/ceph.keyring 2024-03-29T11:18:54.136 DEBUG:teuthology.orchestra.run.smithi084:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-03-29T11:18:54.216 DEBUG:teuthology.orchestra.run.smithi084:> set -ex 2024-03-29T11:18:54.216 DEBUG:teuthology.orchestra.run.smithi084:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-03-29T11:18:54.270 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi179.front.sepia.ceph.com 2024-03-29T11:18:54.270 DEBUG:teuthology.orchestra.run.smithi179:> set -ex 2024-03-29T11:18:54.271 DEBUG:teuthology.orchestra.run.smithi179:> sudo dd of=/etc/ceph/ceph.keyring 2024-03-29T11:18:54.271 DEBUG:teuthology.orchestra.run.smithi179:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-03-29T11:18:54.310 DEBUG:teuthology.orchestra.run.smithi179:> set -ex 2024-03-29T11:18:54.310 DEBUG:teuthology.orchestra.run.smithi179:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-03-29T11:18:54.365 INFO:tasks.ceph:Setting up mon nodes... 2024-03-29T11:18:54.365 INFO:tasks.ceph:Setting up mgr nodes... 2024-03-29T11:18:54.366 DEBUG:teuthology.orchestra.run.smithi084:> sudo mkdir -p /var/lib/ceph/mgr/ceph-y && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.y /var/lib/ceph/mgr/ceph-y/keyring 2024-03-29T11:18:54.420 INFO:teuthology.orchestra.run.smithi084.stdout:creating /var/lib/ceph/mgr/ceph-y/keyring 2024-03-29T11:18:54.423 DEBUG:teuthology.orchestra.run.smithi179:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2024-03-29T11:18:54.493 INFO:teuthology.orchestra.run.smithi179.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2024-03-29T11:18:54.496 INFO:tasks.ceph:Setting up mds nodes... 2024-03-29T11:18:54.496 INFO:tasks.ceph_client:Setting up client nodes... 2024-03-29T11:18:54.496 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-03-29T11:18:54.538 INFO:teuthology.orchestra.run.smithi084.stdout:creating /etc/ceph/ceph.client.0.keyring 2024-03-29T11:18:54.553 DEBUG:teuthology.orchestra.run.smithi179:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.1 /etc/ceph/ceph.client.1.keyring && sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2024-03-29T11:18:54.596 INFO:teuthology.orchestra.run.smithi179.stdout:creating /etc/ceph/ceph.client.1.keyring 2024-03-29T11:18:54.611 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-03-29T11:18:54.612 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi084.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi179.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-03-29T11:18:54.612 DEBUG:teuthology.orchestra.run.smithi084:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2024-03-29T11:18:54.638 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-29T11:18:54.638 INFO:tasks.ceph:role: osd.0 2024-03-29T11:18:54.638 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi084.front.sepia.ceph.com 2024-03-29T11:18:54.638 DEBUG:teuthology.orchestra.run.smithi084:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-03-29T11:18:54.712 INFO:teuthology.orchestra.run.smithi084.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-03-29T11:18:54.712 INFO:teuthology.orchestra.run.smithi084.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-29T11:18:54.712 INFO:teuthology.orchestra.run.smithi084.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-29T11:18:54.712 INFO:teuthology.orchestra.run.smithi084.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-29T11:18:54.712 INFO:teuthology.orchestra.run.smithi084.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-29T11:18:54.712 INFO:teuthology.orchestra.run.smithi084.stdout: = sunit=0 swidth=0 blks 2024-03-29T11:18:54.713 INFO:teuthology.orchestra.run.smithi084.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-29T11:18:54.713 INFO:teuthology.orchestra.run.smithi084.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-29T11:18:54.713 INFO:teuthology.orchestra.run.smithi084.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-29T11:18:54.713 INFO:teuthology.orchestra.run.smithi084.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-29T11:18:55.211 INFO:teuthology.orchestra.run.smithi084.stdout:Discarding blocks...Done. 2024-03-29T11:18:55.212 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi084.front.sepia.ceph.com -o noatime 2024-03-29T11:18:55.213 DEBUG:teuthology.orchestra.run.smithi084:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2024-03-29T11:18:55.348 DEBUG:teuthology.orchestra.run.smithi084:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2024-03-29T11:18:55.380 DEBUG:teuthology.orchestra.run.smithi084:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2024-03-29T11:18:55.447 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-29T11:18:55.448 INFO:tasks.ceph:role: osd.1 2024-03-29T11:18:55.448 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi084.front.sepia.ceph.com 2024-03-29T11:18:55.448 DEBUG:teuthology.orchestra.run.smithi084:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-03-29T11:18:55.522 INFO:teuthology.orchestra.run.smithi084.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-03-29T11:18:55.522 INFO:teuthology.orchestra.run.smithi084.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-29T11:18:55.522 INFO:teuthology.orchestra.run.smithi084.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-29T11:18:55.522 INFO:teuthology.orchestra.run.smithi084.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-29T11:18:55.522 INFO:teuthology.orchestra.run.smithi084.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-29T11:18:55.522 INFO:teuthology.orchestra.run.smithi084.stdout: = sunit=0 swidth=0 blks 2024-03-29T11:18:55.522 INFO:teuthology.orchestra.run.smithi084.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-29T11:18:55.522 INFO:teuthology.orchestra.run.smithi084.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-29T11:18:55.522 INFO:teuthology.orchestra.run.smithi084.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-29T11:18:55.522 INFO:teuthology.orchestra.run.smithi084.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-29T11:18:55.977 INFO:teuthology.orchestra.run.smithi084.stdout:Discarding blocks...Done. 2024-03-29T11:18:55.979 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi084.front.sepia.ceph.com -o noatime 2024-03-29T11:18:55.980 DEBUG:teuthology.orchestra.run.smithi084:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2024-03-29T11:18:56.061 DEBUG:teuthology.orchestra.run.smithi084:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2024-03-29T11:18:56.133 DEBUG:teuthology.orchestra.run.smithi084:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2024-03-29T11:18:56.200 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-29T11:18:56.201 INFO:tasks.ceph:role: osd.2 2024-03-29T11:18:56.201 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi084.front.sepia.ceph.com 2024-03-29T11:18:56.201 DEBUG:teuthology.orchestra.run.smithi084:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-03-29T11:18:56.274 INFO:teuthology.orchestra.run.smithi084.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-03-29T11:18:56.274 INFO:teuthology.orchestra.run.smithi084.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-29T11:18:56.274 INFO:teuthology.orchestra.run.smithi084.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-29T11:18:56.274 INFO:teuthology.orchestra.run.smithi084.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-29T11:18:56.274 INFO:teuthology.orchestra.run.smithi084.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-29T11:18:56.274 INFO:teuthology.orchestra.run.smithi084.stdout: = sunit=0 swidth=0 blks 2024-03-29T11:18:56.275 INFO:teuthology.orchestra.run.smithi084.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-29T11:18:56.275 INFO:teuthology.orchestra.run.smithi084.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-29T11:18:56.275 INFO:teuthology.orchestra.run.smithi084.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-29T11:18:56.275 INFO:teuthology.orchestra.run.smithi084.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-29T11:18:56.779 INFO:teuthology.orchestra.run.smithi084.stdout:Discarding blocks...Done. 2024-03-29T11:18:56.781 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi084.front.sepia.ceph.com -o noatime 2024-03-29T11:18:56.781 DEBUG:teuthology.orchestra.run.smithi084:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2024-03-29T11:18:56.891 DEBUG:teuthology.orchestra.run.smithi084:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2024-03-29T11:18:56.922 DEBUG:teuthology.orchestra.run.smithi084:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2024-03-29T11:18:56.989 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-29T11:18:56.989 INFO:tasks.ceph:role: osd.3 2024-03-29T11:18:56.989 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi084.front.sepia.ceph.com 2024-03-29T11:18:56.990 DEBUG:teuthology.orchestra.run.smithi084:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-03-29T11:18:57.064 INFO:teuthology.orchestra.run.smithi084.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-03-29T11:18:57.064 INFO:teuthology.orchestra.run.smithi084.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-29T11:18:57.064 INFO:teuthology.orchestra.run.smithi084.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-29T11:18:57.064 INFO:teuthology.orchestra.run.smithi084.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-29T11:18:57.064 INFO:teuthology.orchestra.run.smithi084.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-29T11:18:57.064 INFO:teuthology.orchestra.run.smithi084.stdout: = sunit=0 swidth=0 blks 2024-03-29T11:18:57.064 INFO:teuthology.orchestra.run.smithi084.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-29T11:18:57.064 INFO:teuthology.orchestra.run.smithi084.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-29T11:18:57.064 INFO:teuthology.orchestra.run.smithi084.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-29T11:18:57.064 INFO:teuthology.orchestra.run.smithi084.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-29T11:18:57.568 INFO:teuthology.orchestra.run.smithi084.stdout:Discarding blocks...Done. 2024-03-29T11:18:57.570 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi084.front.sepia.ceph.com -o noatime 2024-03-29T11:18:57.570 DEBUG:teuthology.orchestra.run.smithi084:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-3 2024-03-29T11:18:57.701 DEBUG:teuthology.orchestra.run.smithi084:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2024-03-29T11:18:57.735 DEBUG:teuthology.orchestra.run.smithi084:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-29T11:18:57.832 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T11:18:57.854+0000 7f2c9660d640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2024-03-29T11:18:57.833 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T11:18:57.854+0000 7f2c9660d640 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2024-03-29T11:18:57.833 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T11:18:57.854+0000 7f2c9660d640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-03-29T11:18:57.833 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T11:18:57.854+0000 7f2c9660d640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-03-29T11:18:57.833 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T11:18:57.854+0000 7f2c9660d640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-03-29T11:18:57.833 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T11:18:57.855+0000 7f2c9660d640 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2024-03-29T11:19:00.683 DEBUG:teuthology.orchestra.run.smithi084:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-03-29T11:19:00.714 DEBUG:teuthology.orchestra.run.smithi084:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-29T11:19:00.806 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T11:19:00.827+0000 7f95c4014640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2024-03-29T11:19:00.806 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T11:19:00.828+0000 7f95c4014640 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2024-03-29T11:19:00.806 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T11:19:00.828+0000 7f95c4014640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-03-29T11:19:00.806 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T11:19:00.828+0000 7f95c4014640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-03-29T11:19:00.806 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T11:19:00.828+0000 7f95c4014640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-03-29T11:19:00.806 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T11:19:00.828+0000 7f95c4014640 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2024-03-29T11:19:03.652 DEBUG:teuthology.orchestra.run.smithi084:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-03-29T11:19:03.683 DEBUG:teuthology.orchestra.run.smithi084:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-29T11:19:03.774 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T11:19:03.797+0000 7f46fb2f9640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2024-03-29T11:19:03.774 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T11:19:03.797+0000 7f46fb2f9640 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2024-03-29T11:19:03.774 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T11:19:03.797+0000 7f46fb2f9640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-03-29T11:19:03.774 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T11:19:03.797+0000 7f46fb2f9640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-03-29T11:19:03.774 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T11:19:03.797+0000 7f46fb2f9640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-03-29T11:19:03.775 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T11:19:03.798+0000 7f46fb2f9640 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2024-03-29T11:19:06.624 DEBUG:teuthology.orchestra.run.smithi084:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-03-29T11:19:06.655 DEBUG:teuthology.orchestra.run.smithi084:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-29T11:19:06.744 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T11:19:06.767+0000 7f2ac238c640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2024-03-29T11:19:06.744 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T11:19:06.768+0000 7f2ac238c640 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2024-03-29T11:19:06.744 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T11:19:06.768+0000 7f2ac238c640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-03-29T11:19:06.745 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T11:19:06.768+0000 7f2ac238c640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-03-29T11:19:06.745 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T11:19:06.768+0000 7f2ac238c640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-03-29T11:19:06.745 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T11:19:06.768+0000 7f2ac238c640 -1 bluestore(/var/lib/ceph/osd/ceph-3) _read_fsid unparsable uuid 2024-03-29T11:19:09.591 DEBUG:teuthology.orchestra.run.smithi084:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-03-29T11:19:09.623 DEBUG:teuthology.orchestra.run.smithi179:> sudo mkdir -p /var/lib/ceph/osd/ceph-4 2024-03-29T11:19:09.656 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-29T11:19:09.656 INFO:tasks.ceph:role: osd.4 2024-03-29T11:19:09.656 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi179.front.sepia.ceph.com 2024-03-29T11:19:09.656 DEBUG:teuthology.orchestra.run.smithi179:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-03-29T11:19:09.733 INFO:teuthology.orchestra.run.smithi179.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-03-29T11:19:09.733 INFO:teuthology.orchestra.run.smithi179.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-29T11:19:09.733 INFO:teuthology.orchestra.run.smithi179.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-29T11:19:09.733 INFO:teuthology.orchestra.run.smithi179.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-29T11:19:09.733 INFO:teuthology.orchestra.run.smithi179.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-29T11:19:09.733 INFO:teuthology.orchestra.run.smithi179.stdout: = sunit=0 swidth=0 blks 2024-03-29T11:19:09.734 INFO:teuthology.orchestra.run.smithi179.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-29T11:19:09.734 INFO:teuthology.orchestra.run.smithi179.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-29T11:19:09.734 INFO:teuthology.orchestra.run.smithi179.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-29T11:19:09.734 INFO:teuthology.orchestra.run.smithi179.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-29T11:19:10.249 INFO:teuthology.orchestra.run.smithi179.stdout:Discarding blocks...Done. 2024-03-29T11:19:10.251 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi179.front.sepia.ceph.com -o noatime 2024-03-29T11:19:10.251 DEBUG:teuthology.orchestra.run.smithi179:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-4 2024-03-29T11:19:10.373 DEBUG:teuthology.orchestra.run.smithi179:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-4 2024-03-29T11:19:10.409 DEBUG:teuthology.orchestra.run.smithi179:> sudo mkdir -p /var/lib/ceph/osd/ceph-5 2024-03-29T11:19:10.478 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-29T11:19:10.478 INFO:tasks.ceph:role: osd.5 2024-03-29T11:19:10.478 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi179.front.sepia.ceph.com 2024-03-29T11:19:10.478 DEBUG:teuthology.orchestra.run.smithi179:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-03-29T11:19:10.555 INFO:teuthology.orchestra.run.smithi179.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-03-29T11:19:10.555 INFO:teuthology.orchestra.run.smithi179.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-29T11:19:10.555 INFO:teuthology.orchestra.run.smithi179.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-29T11:19:10.555 INFO:teuthology.orchestra.run.smithi179.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-29T11:19:10.555 INFO:teuthology.orchestra.run.smithi179.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-29T11:19:10.555 INFO:teuthology.orchestra.run.smithi179.stdout: = sunit=0 swidth=0 blks 2024-03-29T11:19:10.555 INFO:teuthology.orchestra.run.smithi179.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-29T11:19:10.555 INFO:teuthology.orchestra.run.smithi179.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-29T11:19:10.555 INFO:teuthology.orchestra.run.smithi179.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-29T11:19:10.555 INFO:teuthology.orchestra.run.smithi179.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-29T11:19:11.022 INFO:teuthology.orchestra.run.smithi179.stdout:Discarding blocks...Done. 2024-03-29T11:19:11.024 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi179.front.sepia.ceph.com -o noatime 2024-03-29T11:19:11.024 DEBUG:teuthology.orchestra.run.smithi179:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-5 2024-03-29T11:19:11.126 DEBUG:teuthology.orchestra.run.smithi179:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-5 2024-03-29T11:19:11.160 DEBUG:teuthology.orchestra.run.smithi179:> sudo mkdir -p /var/lib/ceph/osd/ceph-6 2024-03-29T11:19:11.229 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-29T11:19:11.229 INFO:tasks.ceph:role: osd.6 2024-03-29T11:19:11.229 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi179.front.sepia.ceph.com 2024-03-29T11:19:11.229 DEBUG:teuthology.orchestra.run.smithi179:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-03-29T11:19:11.305 INFO:teuthology.orchestra.run.smithi179.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-03-29T11:19:11.305 INFO:teuthology.orchestra.run.smithi179.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-29T11:19:11.305 INFO:teuthology.orchestra.run.smithi179.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-29T11:19:11.305 INFO:teuthology.orchestra.run.smithi179.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-29T11:19:11.305 INFO:teuthology.orchestra.run.smithi179.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-29T11:19:11.305 INFO:teuthology.orchestra.run.smithi179.stdout: = sunit=0 swidth=0 blks 2024-03-29T11:19:11.305 INFO:teuthology.orchestra.run.smithi179.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-29T11:19:11.305 INFO:teuthology.orchestra.run.smithi179.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-29T11:19:11.305 INFO:teuthology.orchestra.run.smithi179.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-29T11:19:11.305 INFO:teuthology.orchestra.run.smithi179.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-29T11:19:11.786 INFO:teuthology.orchestra.run.smithi179.stdout:Discarding blocks...Done. 2024-03-29T11:19:11.789 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi179.front.sepia.ceph.com -o noatime 2024-03-29T11:19:11.789 DEBUG:teuthology.orchestra.run.smithi179:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-6 2024-03-29T11:19:11.922 DEBUG:teuthology.orchestra.run.smithi179:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-6 2024-03-29T11:19:11.959 DEBUG:teuthology.orchestra.run.smithi179:> sudo mkdir -p /var/lib/ceph/osd/ceph-7 2024-03-29T11:19:12.028 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-29T11:19:12.028 INFO:tasks.ceph:role: osd.7 2024-03-29T11:19:12.028 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi179.front.sepia.ceph.com 2024-03-29T11:19:12.028 DEBUG:teuthology.orchestra.run.smithi179:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-03-29T11:19:12.104 INFO:teuthology.orchestra.run.smithi179.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-03-29T11:19:12.104 INFO:teuthology.orchestra.run.smithi179.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-29T11:19:12.104 INFO:teuthology.orchestra.run.smithi179.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-29T11:19:12.104 INFO:teuthology.orchestra.run.smithi179.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-29T11:19:12.104 INFO:teuthology.orchestra.run.smithi179.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-29T11:19:12.104 INFO:teuthology.orchestra.run.smithi179.stdout: = sunit=0 swidth=0 blks 2024-03-29T11:19:12.104 INFO:teuthology.orchestra.run.smithi179.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-29T11:19:12.104 INFO:teuthology.orchestra.run.smithi179.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-29T11:19:12.104 INFO:teuthology.orchestra.run.smithi179.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-29T11:19:12.104 INFO:teuthology.orchestra.run.smithi179.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-29T11:19:12.627 INFO:teuthology.orchestra.run.smithi179.stdout:Discarding blocks...Done. 2024-03-29T11:19:12.629 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi179.front.sepia.ceph.com -o noatime 2024-03-29T11:19:12.630 DEBUG:teuthology.orchestra.run.smithi179:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-7 2024-03-29T11:19:12.763 DEBUG:teuthology.orchestra.run.smithi179:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-7 2024-03-29T11:19:12.800 DEBUG:teuthology.orchestra.run.smithi179:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 4 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-29T11:19:12.910 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:12.933+0000 7fb862c0e640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-4/keyring: can't open /var/lib/ceph/osd/ceph-4/keyring: (2) No such file or directory 2024-03-29T11:19:12.911 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:12.934+0000 7fb862c0e640 -1 created new key in keyring /var/lib/ceph/osd/ceph-4/keyring 2024-03-29T11:19:12.911 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:12.934+0000 7fb862c0e640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-03-29T11:19:12.911 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:12.934+0000 7fb862c0e640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-03-29T11:19:12.911 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:12.934+0000 7fb862c0e640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-03-29T11:19:12.911 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:12.934+0000 7fb862c0e640 -1 bluestore(/var/lib/ceph/osd/ceph-4) _read_fsid unparsable uuid 2024-03-29T11:19:15.765 DEBUG:teuthology.orchestra.run.smithi179:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-03-29T11:19:15.796 DEBUG:teuthology.orchestra.run.smithi179:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 5 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-29T11:19:15.888 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:15.911+0000 7fd8c770f640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-5/keyring: can't open /var/lib/ceph/osd/ceph-5/keyring: (2) No such file or directory 2024-03-29T11:19:15.889 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:15.912+0000 7fd8c770f640 -1 created new key in keyring /var/lib/ceph/osd/ceph-5/keyring 2024-03-29T11:19:15.889 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:15.912+0000 7fd8c770f640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-03-29T11:19:15.889 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:15.912+0000 7fd8c770f640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-03-29T11:19:15.889 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:15.912+0000 7fd8c770f640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-03-29T11:19:15.889 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:15.912+0000 7fd8c770f640 -1 bluestore(/var/lib/ceph/osd/ceph-5) _read_fsid unparsable uuid 2024-03-29T11:19:18.746 DEBUG:teuthology.orchestra.run.smithi179:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-03-29T11:19:18.777 DEBUG:teuthology.orchestra.run.smithi179:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 6 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-29T11:19:18.870 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:18.893+0000 7f6d6660d640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-6/keyring: can't open /var/lib/ceph/osd/ceph-6/keyring: (2) No such file or directory 2024-03-29T11:19:18.871 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:18.894+0000 7f6d6660d640 -1 created new key in keyring /var/lib/ceph/osd/ceph-6/keyring 2024-03-29T11:19:18.871 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:18.894+0000 7f6d6660d640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-03-29T11:19:18.871 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:18.894+0000 7f6d6660d640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-03-29T11:19:18.871 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:18.894+0000 7f6d6660d640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-03-29T11:19:18.871 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:18.894+0000 7f6d6660d640 -1 bluestore(/var/lib/ceph/osd/ceph-6) _read_fsid unparsable uuid 2024-03-29T11:19:21.726 DEBUG:teuthology.orchestra.run.smithi179:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-03-29T11:19:21.757 DEBUG:teuthology.orchestra.run.smithi179:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 7 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-29T11:19:21.847 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:21.870+0000 7fe813c08640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-7/keyring: can't open /var/lib/ceph/osd/ceph-7/keyring: (2) No such file or directory 2024-03-29T11:19:21.848 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:21.871+0000 7fe813c08640 -1 created new key in keyring /var/lib/ceph/osd/ceph-7/keyring 2024-03-29T11:19:21.848 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:21.871+0000 7fe813c08640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-03-29T11:19:21.848 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:21.871+0000 7fe813c08640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-03-29T11:19:21.848 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:21.871+0000 7fe813c08640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-03-29T11:19:21.848 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:21.871+0000 7fe813c08640 -1 bluestore(/var/lib/ceph/osd/ceph-7) _read_fsid unparsable uuid 2024-03-29T11:19:24.690 DEBUG:teuthology.orchestra.run.smithi179:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-03-29T11:19:24.721 INFO:tasks.ceph:Reading keys from all nodes... 2024-03-29T11:19:24.721 DEBUG:teuthology.orchestra.run.smithi084:> set -ex 2024-03-29T11:19:24.722 DEBUG:teuthology.orchestra.run.smithi084:> sudo dd if=/var/lib/ceph/mgr/ceph-y/keyring of=/dev/stdout 2024-03-29T11:19:24.753 DEBUG:teuthology.orchestra.run.smithi084:> set -ex 2024-03-29T11:19:24.753 DEBUG:teuthology.orchestra.run.smithi084:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2024-03-29T11:19:24.823 DEBUG:teuthology.orchestra.run.smithi084:> set -ex 2024-03-29T11:19:24.823 DEBUG:teuthology.orchestra.run.smithi084:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2024-03-29T11:19:24.893 DEBUG:teuthology.orchestra.run.smithi084:> set -ex 2024-03-29T11:19:24.893 DEBUG:teuthology.orchestra.run.smithi084:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2024-03-29T11:19:24.961 DEBUG:teuthology.orchestra.run.smithi084:> set -ex 2024-03-29T11:19:24.961 DEBUG:teuthology.orchestra.run.smithi084:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2024-03-29T11:19:25.029 DEBUG:teuthology.orchestra.run.smithi179:> set -ex 2024-03-29T11:19:25.029 DEBUG:teuthology.orchestra.run.smithi179:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2024-03-29T11:19:25.060 DEBUG:teuthology.orchestra.run.smithi179:> set -ex 2024-03-29T11:19:25.061 DEBUG:teuthology.orchestra.run.smithi179:> sudo dd if=/var/lib/ceph/osd/ceph-4/keyring of=/dev/stdout 2024-03-29T11:19:25.130 DEBUG:teuthology.orchestra.run.smithi179:> set -ex 2024-03-29T11:19:25.130 DEBUG:teuthology.orchestra.run.smithi179:> sudo dd if=/var/lib/ceph/osd/ceph-5/keyring of=/dev/stdout 2024-03-29T11:19:25.200 DEBUG:teuthology.orchestra.run.smithi179:> set -ex 2024-03-29T11:19:25.200 DEBUG:teuthology.orchestra.run.smithi179:> sudo dd if=/var/lib/ceph/osd/ceph-6/keyring of=/dev/stdout 2024-03-29T11:19:25.268 DEBUG:teuthology.orchestra.run.smithi179:> set -ex 2024-03-29T11:19:25.268 DEBUG:teuthology.orchestra.run.smithi179:> sudo dd if=/var/lib/ceph/osd/ceph-7/keyring of=/dev/stdout 2024-03-29T11:19:25.335 DEBUG:teuthology.orchestra.run.smithi084:> set -ex 2024-03-29T11:19:25.335 DEBUG:teuthology.orchestra.run.smithi084:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2024-03-29T11:19:25.352 DEBUG:teuthology.orchestra.run.smithi179:> set -ex 2024-03-29T11:19:25.352 DEBUG:teuthology.orchestra.run.smithi179:> dd if=/etc/ceph/ceph.client.1.keyring of=/dev/stdout 2024-03-29T11:19:25.390 INFO:tasks.ceph:Adding keys to all mons... 2024-03-29T11:19:25.391 DEBUG:teuthology.orchestra.run.smithi084:> sudo tee -a /etc/ceph/ceph.keyring 2024-03-29T11:19:25.395 DEBUG:teuthology.orchestra.run.smithi179:> sudo tee -a /etc/ceph/ceph.keyring 2024-03-29T11:19:25.434 INFO:teuthology.orchestra.run.smithi084.stdout:[mgr.y] 2024-03-29T11:19:25.434 INFO:teuthology.orchestra.run.smithi084.stdout: key = AQCeowZmiRyCGhAAmlInlw0PnQFEVAboEcrFPQ== 2024-03-29T11:19:25.434 INFO:teuthology.orchestra.run.smithi084.stdout:[osd.0] 2024-03-29T11:19:25.434 INFO:teuthology.orchestra.run.smithi084.stdout: key = AQChowZmAzAMMxAAVCzBrwXc1PxtZdIplZa1ow== 2024-03-29T11:19:25.434 INFO:teuthology.orchestra.run.smithi084.stdout:[osd.1] 2024-03-29T11:19:25.434 INFO:teuthology.orchestra.run.smithi084.stdout: key = AQCkowZmTn1zMRAAyDpl9Ri2JHo9pB87fMTJpA== 2024-03-29T11:19:25.434 INFO:teuthology.orchestra.run.smithi084.stdout:[osd.2] 2024-03-29T11:19:25.434 INFO:teuthology.orchestra.run.smithi084.stdout: key = AQCnowZm3XySLxAAkrZxarfbZxQeB4nCjRrAJQ== 2024-03-29T11:19:25.434 INFO:teuthology.orchestra.run.smithi084.stdout:[osd.3] 2024-03-29T11:19:25.435 INFO:teuthology.orchestra.run.smithi084.stdout: key = AQCqowZmE5DQLRAA9pmNgXhnUQxvBrzjWCwEew== 2024-03-29T11:19:25.435 INFO:teuthology.orchestra.run.smithi084.stdout:[mgr.x] 2024-03-29T11:19:25.435 INFO:teuthology.orchestra.run.smithi084.stdout: key = AQCeowZmJQPcHhAA7+wPyPkUx6lzyvx4g8GFvg== 2024-03-29T11:19:25.435 INFO:teuthology.orchestra.run.smithi084.stdout:[osd.4] 2024-03-29T11:19:25.435 INFO:teuthology.orchestra.run.smithi084.stdout: key = AQCwowZmSgi3NxAAI1ddyk1DDLdFYa4pZKykrQ== 2024-03-29T11:19:25.435 INFO:teuthology.orchestra.run.smithi084.stdout:[osd.5] 2024-03-29T11:19:25.435 INFO:teuthology.orchestra.run.smithi084.stdout: key = AQCzowZmrmRoNhAA7PmH4dJk19rh+S6UCDg4+w== 2024-03-29T11:19:25.435 INFO:teuthology.orchestra.run.smithi084.stdout:[osd.6] 2024-03-29T11:19:25.435 INFO:teuthology.orchestra.run.smithi084.stdout: key = AQC2owZm7sVTNRAALus9wbWxlkhCzUkaT3I+nQ== 2024-03-29T11:19:25.435 INFO:teuthology.orchestra.run.smithi084.stdout:[osd.7] 2024-03-29T11:19:25.435 INFO:teuthology.orchestra.run.smithi084.stdout: key = AQC5owZmVTT3MxAAbENpvm4heSiZEXuUU5TFmg== 2024-03-29T11:19:25.435 INFO:teuthology.orchestra.run.smithi084.stdout:[client.0] 2024-03-29T11:19:25.435 INFO:teuthology.orchestra.run.smithi084.stdout: key = AQCeowZmNkWKIRAAFe/94KqsKd/29Kvu8wcDXA== 2024-03-29T11:19:25.435 INFO:teuthology.orchestra.run.smithi084.stdout:[client.1] 2024-03-29T11:19:25.435 INFO:teuthology.orchestra.run.smithi084.stdout: key = AQCeowZmXET/JBAAG7KH7rwA1LTv+XCaygdsWA== 2024-03-29T11:19:25.456 INFO:teuthology.orchestra.run.smithi179.stdout:[mgr.y] 2024-03-29T11:19:25.456 INFO:teuthology.orchestra.run.smithi179.stdout: key = AQCeowZmiRyCGhAAmlInlw0PnQFEVAboEcrFPQ== 2024-03-29T11:19:25.456 INFO:teuthology.orchestra.run.smithi179.stdout:[osd.0] 2024-03-29T11:19:25.457 INFO:teuthology.orchestra.run.smithi179.stdout: key = AQChowZmAzAMMxAAVCzBrwXc1PxtZdIplZa1ow== 2024-03-29T11:19:25.457 INFO:teuthology.orchestra.run.smithi179.stdout:[osd.1] 2024-03-29T11:19:25.457 INFO:teuthology.orchestra.run.smithi179.stdout: key = AQCkowZmTn1zMRAAyDpl9Ri2JHo9pB87fMTJpA== 2024-03-29T11:19:25.457 INFO:teuthology.orchestra.run.smithi179.stdout:[osd.2] 2024-03-29T11:19:25.457 INFO:teuthology.orchestra.run.smithi179.stdout: key = AQCnowZm3XySLxAAkrZxarfbZxQeB4nCjRrAJQ== 2024-03-29T11:19:25.457 INFO:teuthology.orchestra.run.smithi179.stdout:[osd.3] 2024-03-29T11:19:25.457 INFO:teuthology.orchestra.run.smithi179.stdout: key = AQCqowZmE5DQLRAA9pmNgXhnUQxvBrzjWCwEew== 2024-03-29T11:19:25.457 INFO:teuthology.orchestra.run.smithi179.stdout:[mgr.x] 2024-03-29T11:19:25.457 INFO:teuthology.orchestra.run.smithi179.stdout: key = AQCeowZmJQPcHhAA7+wPyPkUx6lzyvx4g8GFvg== 2024-03-29T11:19:25.457 INFO:teuthology.orchestra.run.smithi179.stdout:[osd.4] 2024-03-29T11:19:25.457 INFO:teuthology.orchestra.run.smithi179.stdout: key = AQCwowZmSgi3NxAAI1ddyk1DDLdFYa4pZKykrQ== 2024-03-29T11:19:25.457 INFO:teuthology.orchestra.run.smithi179.stdout:[osd.5] 2024-03-29T11:19:25.457 INFO:teuthology.orchestra.run.smithi179.stdout: key = AQCzowZmrmRoNhAA7PmH4dJk19rh+S6UCDg4+w== 2024-03-29T11:19:25.457 INFO:teuthology.orchestra.run.smithi179.stdout:[osd.6] 2024-03-29T11:19:25.457 INFO:teuthology.orchestra.run.smithi179.stdout: key = AQC2owZm7sVTNRAALus9wbWxlkhCzUkaT3I+nQ== 2024-03-29T11:19:25.458 INFO:teuthology.orchestra.run.smithi179.stdout:[osd.7] 2024-03-29T11:19:25.458 INFO:teuthology.orchestra.run.smithi179.stdout: key = AQC5owZmVTT3MxAAbENpvm4heSiZEXuUU5TFmg== 2024-03-29T11:19:25.458 INFO:teuthology.orchestra.run.smithi179.stdout:[client.0] 2024-03-29T11:19:25.458 INFO:teuthology.orchestra.run.smithi179.stdout: key = AQCeowZmNkWKIRAAFe/94KqsKd/29Kvu8wcDXA== 2024-03-29T11:19:25.458 INFO:teuthology.orchestra.run.smithi179.stdout:[client.1] 2024-03-29T11:19:25.458 INFO:teuthology.orchestra.run.smithi179.stdout: key = AQCeowZmXET/JBAAG7KH7rwA1LTv+XCaygdsWA== 2024-03-29T11:19:25.459 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-29T11:19:25.479 DEBUG:teuthology.orchestra.run.smithi179:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-29T11:19:25.551 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:19:25.553 DEBUG:teuthology.orchestra.run.smithi179:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:19:25.605 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:19:25.607 DEBUG:teuthology.orchestra.run.smithi179:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:19:25.659 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:19:25.661 DEBUG:teuthology.orchestra.run.smithi179:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:19:25.712 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:19:25.714 DEBUG:teuthology.orchestra.run.smithi179:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:19:25.766 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-29T11:19:25.768 DEBUG:teuthology.orchestra.run.smithi179:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-29T11:19:25.819 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:19:25.821 DEBUG:teuthology.orchestra.run.smithi179:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:19:25.873 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:19:25.875 DEBUG:teuthology.orchestra.run.smithi179:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:19:25.927 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:19:25.929 DEBUG:teuthology.orchestra.run.smithi179:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:19:25.980 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:19:25.982 DEBUG:teuthology.orchestra.run.smithi179:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:19:26.034 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-29T11:19:26.036 DEBUG:teuthology.orchestra.run.smithi179:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-29T11:19:26.092 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-29T11:19:26.095 DEBUG:teuthology.orchestra.run.smithi179:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-29T11:19:26.149 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-03-29T11:19:26.149 DEBUG:teuthology.orchestra.run.smithi084:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2024-03-29T11:19:26.175 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-03-29T11:19:26.272 DEBUG:teuthology.orchestra.run.smithi084:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2024-03-29T11:19:26.298 DEBUG:teuthology.orchestra.run.smithi084:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2024-03-29T11:19:26.365 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-03-29T11:19:26.461 DEBUG:teuthology.orchestra.run.smithi084:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2024-03-29T11:19:26.490 DEBUG:teuthology.orchestra.run.smithi179:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2024-03-29T11:19:26.518 DEBUG:teuthology.orchestra.run.smithi179:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-03-29T11:19:26.616 DEBUG:teuthology.orchestra.run.smithi179:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2024-03-29T11:19:26.643 DEBUG:teuthology.orchestra.run.smithi084:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-03-29T11:19:26.645 DEBUG:teuthology.orchestra.run.smithi179:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-03-29T11:19:26.698 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2024-03-29T11:19:26.699 INFO:tasks.ceph.mon.a:Restarting daemon 2024-03-29T11:19:26.699 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-03-29T11:19:26.701 INFO:tasks.ceph.mon.a:Started 2024-03-29T11:19:26.701 INFO:tasks.ceph.mon.c:Restarting daemon 2024-03-29T11:19:26.701 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2024-03-29T11:19:26.703 INFO:tasks.ceph.mon.c:Started 2024-03-29T11:19:26.703 INFO:tasks.ceph.mon.b:Restarting daemon 2024-03-29T11:19:26.703 DEBUG:teuthology.orchestra.run.smithi179:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2024-03-29T11:19:26.741 INFO:tasks.ceph.mon.b:Started 2024-03-29T11:19:26.741 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2024-03-29T11:19:26.741 INFO:tasks.ceph.mgr.y:Restarting daemon 2024-03-29T11:19:26.741 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i y 2024-03-29T11:19:26.744 INFO:tasks.ceph.mgr.y:Started 2024-03-29T11:19:26.744 INFO:tasks.ceph.mgr.x:Restarting daemon 2024-03-29T11:19:26.744 DEBUG:teuthology.orchestra.run.smithi179:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2024-03-29T11:19:26.746 INFO:tasks.ceph.mgr.x:Started 2024-03-29T11:19:26.746 INFO:tasks.ceph:Setting crush tunables to default 2024-03-29T11:19:26.746 DEBUG:teuthology.orchestra.run.smithi084:> sudo ceph --cluster ceph osd crush tunables default 2024-03-29T11:19:26.848 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T11:19:26.871+0000 7f1fa99c6640 10 monclient: get_monmap_and_config 2024-03-29T11:19:26.848 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T11:19:26.871+0000 7f1fa99c6640 10 monclient: build_initial_monmap 2024-03-29T11:19:26.848 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T11:19:26.871+0000 7f1fa99c6640 10 monclient: monmap: 2024-03-29T11:19:26.848 INFO:teuthology.orchestra.run.smithi084.stderr:epoch 0 2024-03-29T11:19:26.848 INFO:teuthology.orchestra.run.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:19:26.848 INFO:teuthology.orchestra.run.smithi084.stderr:last_changed 2024-03-29T11:19:26.872254+0000 2024-03-29T11:19:26.848 INFO:teuthology.orchestra.run.smithi084.stderr:created 2024-03-29T11:19:26.872254+0000 2024-03-29T11:19:26.848 INFO:teuthology.orchestra.run.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-29T11:19:26.848 INFO:teuthology.orchestra.run.smithi084.stderr:election_strategy: 1 2024-03-29T11:19:26.848 INFO:teuthology.orchestra.run.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-29T11:19:26.848 INFO:teuthology.orchestra.run.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.noname-c 2024-03-29T11:19:26.848 INFO:teuthology.orchestra.run.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-b 2024-03-29T11:19:26.849 INFO:teuthology.orchestra.run.smithi084.stderr: 2024-03-29T11:19:26.850 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T11:19:26.871+0000 7f1fa99c6640 1 Processor -- start 2024-03-29T11:19:26.850 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T11:19:26.872+0000 7f1fa99c6640 1 -- start start 2024-03-29T11:19:26.850 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T11:19:26.872+0000 7f1fa99c6640 10 monclient: init 2024-03-29T11:19:26.850 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T11:19:26.872+0000 7f1fa99c6640 10 monclient: _reopen_session rank -1 2024-03-29T11:19:26.850 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T11:19:26.872+0000 7f1fa99c6640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:19:26.850 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T11:19:26.872+0000 7f1fa99c6640 10 monclient(hunting): picked mon.noname-c con 0x7f1fa40b8240 addr v1:172.21.15.179:6789/0 2024-03-29T11:19:26.850 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T11:19:26.872+0000 7f1fa99c6640 10 monclient(hunting): picked mon.noname-b con 0x7f1fa40b1f60 addr v1:172.21.15.84:6790/0 2024-03-29T11:19:26.850 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T11:19:26.872+0000 7f1fa99c6640 10 monclient(hunting): picked mon.noname-a con 0x7f1fa409c8c0 addr v1:172.21.15.84:6789/0 2024-03-29T11:19:26.851 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T11:19:26.872+0000 7f1fa99c6640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1fa409bd80 con 0x7f1fa409c8c0 2024-03-29T11:19:26.851 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T11:19:26.872+0000 7f1fa99c6640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1fa409b960 con 0x7f1fa40b8240 2024-03-29T11:19:26.851 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T11:19:26.872+0000 7f1fa99c6640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1fa40b2ec0 con 0x7f1fa40b1f60 2024-03-29T11:19:26.851 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T11:19:26.872+0000 7f1fa99c6640 10 monclient(hunting): _renew_subs 2024-03-29T11:19:26.851 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T11:19:26.872+0000 7f1fa99c6640 10 monclient(hunting): authenticate will time out at 828.166687s 2024-03-29T11:19:26.851 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T11:19:26.872+0000 7f1fa27fc640 1 --1- >> v1:172.21.15.84:6790/0 conn(0x7f1fa40b1f60 0x7f1fa409c1b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.84:6790/0 says I am v1:172.21.15.84:52392/0 (socket says 172.21.15.84:52392) 2024-03-29T11:19:26.851 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T11:19:26.872+0000 7f1fa27fc640 1 -- 172.21.15.84:0/3246226570 learned_addr learned my addr 172.21.15.84:0/3246226570 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-29T11:19:26.851 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T11:19:26.872+0000 7f1fa37fe640 1 --1- 172.21.15.84:0/3246226570 >> v1:172.21.15.84:6789/0 conn(0x7f1fa409c8c0 0x7f1fa414d900 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.84:6789/0 says I am v1:172.21.15.84:34310/0 (socket says 172.21.15.84:34310) 2024-03-29T11:19:29.850 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T11:19:29.872+0000 7f1fa0ff9640 10 monclient(hunting): tick 2024-03-29T11:19:29.850 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T11:19:29.872+0000 7f1fa0ff9640 10 monclient(hunting): _check_auth_tickets 2024-03-29T11:19:29.850 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T11:19:29.872+0000 7f1fa0ff9640 1 monclient(hunting): continuing hunt 2024-03-29T11:19:29.850 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T11:19:29.872+0000 7f1fa0ff9640 10 monclient(hunting): _reopen_session rank -1 2024-03-29T11:19:29.850 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T11:19:29.872+0000 7f1fa0ff9640 1 -- 172.21.15.84:0/3246226570 >> v1:172.21.15.84:6790/0 conn(0x7f1fa40b1f60 legacy=0x7f1fa409c1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:29.850 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T11:19:29.872+0000 7f1fa0ff9640 1 -- 172.21.15.84:0/3246226570 >> v1:172.21.15.179:6789/0 conn(0x7f1fa40b8240 legacy=0x7f1fa413f450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:29.851 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T11:19:29.872+0000 7f1fa0ff9640 1 -- 172.21.15.84:0/3246226570 >> v1:172.21.15.84:6789/0 conn(0x7f1fa409c8c0 legacy=0x7f1fa414d900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:29.851 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T11:19:29.872+0000 7f1fa0ff9640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:19:29.851 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T11:19:29.872+0000 7f1fa0ff9640 10 monclient(hunting): picked mon.noname-b con 0x7f1f840036e0 addr v1:172.21.15.84:6790/0 2024-03-29T11:19:29.851 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T11:19:29.872+0000 7f1fa0ff9640 10 monclient(hunting): picked mon.noname-a con 0x7f1f840072d0 addr v1:172.21.15.84:6789/0 2024-03-29T11:19:29.851 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T11:19:29.872+0000 7f1fa0ff9640 10 monclient(hunting): picked mon.noname-c con 0x7f1fa40b8240 addr v1:172.21.15.179:6789/0 2024-03-29T11:19:29.851 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T11:19:29.872+0000 7f1fa0ff9640 1 -- 172.21.15.84:0/3246226570 --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1fa413fb60 con 0x7f1f840072d0 2024-03-29T11:19:29.851 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T11:19:29.872+0000 7f1fa0ff9640 1 -- 172.21.15.84:0/3246226570 --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1f8400f8b0 con 0x7f1fa40b8240 2024-03-29T11:19:29.851 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T11:19:29.872+0000 7f1fa0ff9640 1 -- 172.21.15.84:0/3246226570 --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1f84010b10 con 0x7f1f840036e0 2024-03-29T11:19:29.851 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T11:19:29.872+0000 7f1fa0ff9640 10 monclient(hunting): _renew_subs 2024-03-29T11:19:32.025 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T11:19:32.043+0000 7f1fa1ffb640 1 -- 172.21.15.84:0/3246226570 <== mon.0 v1:172.21.15.84:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3342770064 0 0) 0x7f1fa413fb60 con 0x7f1f840072d0 2024-03-29T11:19:32.025 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T11:19:32.043+0000 7f1fa1ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:19:32.025 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T11:19:32.043+0000 7f1fa1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:32.025 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T11:19:32.043+0000 7f1fa1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:32.025 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T11:19:32.043+0000 7f1fa1ffb640 10 monclient(hunting): my global_id is 4101 2024-03-29T11:19:32.025 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T11:19:32.043+0000 7f1fa1ffb640 1 -- 172.21.15.84:0/3246226570 --> v1:172.21.15.84:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1f90003520 con 0x7f1f840072d0 2024-03-29T11:19:32.025 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T11:19:32.044+0000 7f1fa1ffb640 1 -- 172.21.15.84:0/3246226570 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 641664643 0 0) 0x7f1f84010b10 con 0x7f1f840036e0 2024-03-29T11:19:32.025 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T11:19:32.044+0000 7f1fa1ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:19:32.025 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T11:19:32.044+0000 7f1fa1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:32.026 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T11:19:32.044+0000 7f1fa1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:32.026 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T11:19:32.044+0000 7f1fa1ffb640 10 monclient(hunting): my global_id is 4106 2024-03-29T11:19:32.026 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T11:19:32.044+0000 7f1fa1ffb640 1 -- 172.21.15.84:0/3246226570 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1fa413fb60 con 0x7f1f840036e0 2024-03-29T11:19:32.026 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T11:19:32.044+0000 7f1fa1ffb640 1 -- 172.21.15.84:0/3246226570 <== mon.2 v1:172.21.15.84:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2939571116 0 0) 0x7f1fa413fb60 con 0x7f1f840036e0 2024-03-29T11:19:32.026 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T11:19:32.044+0000 7f1fa1ffb640 1 -- 172.21.15.84:0/3246226570 >> v1:172.21.15.179:6789/0 conn(0x7f1fa40b8240 legacy=0x7f1f8400d030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:32.026 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T11:19:32.044+0000 7f1fa1ffb640 1 -- 172.21.15.84:0/3246226570 >> v1:172.21.15.84:6789/0 conn(0x7f1f840072d0 legacy=0x7f1f84009720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:32.026 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T11:19:32.044+0000 7f1fa1ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:19:32.026 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T11:19:32.044+0000 7f1fa1ffb640 1 monclient: found mon.noname-b 2024-03-29T11:19:32.026 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T11:19:32.044+0000 7f1fa1ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:19:32.026 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T11:19:32.044+0000 7f1fa1ffb640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.84:6790/0 2024-03-29T11:19:32.026 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T11:19:32.044+0000 7f1fa1ffb640 1 -- 172.21.15.84:0/3246226570 --> v1:172.21.15.84:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f84011d70 con 0x7f1f840036e0 2024-03-29T11:19:32.026 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T11:19:32.044+0000 7f1fa1ffb640 10 monclient: _finish_auth 0 2024-03-29T11:19:32.027 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T11:19:32.044+0000 7f1fa1ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:19:32.027 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T11:19:32.044+0000 7f1fa1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:19:32.027 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T11:19:32.044+0000 7f1fa1ffb640 1 -- 172.21.15.84:0/3246226570 <== mon.2 v1:172.21.15.84:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4236868694 0 0) 0x7f1f94001610 con 0x7f1f840036e0 2024-03-29T11:19:32.027 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T11:19:32.044+0000 7f1fa1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:32.027 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T11:19:32.044+0000 7f1fa1ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:19:32.027 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T11:19:32.044+0000 7f1fa1ffb640 10 monclient: dump: 2024-03-29T11:19:32.027 INFO:teuthology.orchestra.run.smithi084.stderr:epoch 1 2024-03-29T11:19:32.027 INFO:teuthology.orchestra.run.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:19:32.027 INFO:teuthology.orchestra.run.smithi084.stderr:last_changed 2024-03-29T11:18:53.864025+0000 2024-03-29T11:19:32.027 INFO:teuthology.orchestra.run.smithi084.stderr:created 2024-03-29T11:18:53.864025+0000 2024-03-29T11:19:32.027 INFO:teuthology.orchestra.run.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:32.027 INFO:teuthology.orchestra.run.smithi084.stderr:election_strategy: 1 2024-03-29T11:19:32.027 INFO:teuthology.orchestra.run.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T11:19:32.028 INFO:teuthology.orchestra.run.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T11:19:32.028 INFO:teuthology.orchestra.run.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T11:19:32.028 INFO:teuthology.orchestra.run.smithi084.stderr: 2024-03-29T11:19:32.028 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T11:19:32.044+0000 7f1fa99c6640 5 monclient: authenticate success, global_id 4106 2024-03-29T11:19:32.028 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T11:19:32.044+0000 7f1fa99c6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:19:32.028 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T11:19:32.044+0000 7f1fa2ffd640 1 -- 172.21.15.84:0/3246226570 >> v1:172.21.15.84:6790/0 conn(0x7f1f840036e0 legacy=0x7f1f84005b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-03-29T11:19:32.028 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T11:19:32.044+0000 7f1fa2ffd640 1 -- 172.21.15.84:0/3246226570 >> v1:172.21.15.84:6790/0 conn(0x7f1f840036e0 legacy=0x7f1f84005b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-29T11:19:32.028 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T11:19:32.044+0000 7f1fa2ffd640 1 --1- 172.21.15.84:0/3246226570 >> v1:172.21.15.84:6790/0 conn(0x7f1f840036e0 0x7f1f84005b30 :-1 s=OPENED pgs=11 cs=1 l=1).handle_message read tag failed 2024-03-29T11:19:32.028 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T11:19:32.044+0000 7f1fa2ffd640 1 --1- 172.21.15.84:0/3246226570 >> v1:172.21.15.84:6790/0 conn(0x7f1f840036e0 0x7f1f84005b30 :-1 s=OPENED pgs=11 cs=1 l=1).fault on lossy channel, failing 2024-03-29T11:19:32.028 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T11:19:32.044+0000 7f1fa1ffb640 10 monclient: ms_handle_reset current mon v1:172.21.15.84:6790/0 2024-03-29T11:19:32.028 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T11:19:32.044+0000 7f1fa1ffb640 10 monclient: _reopen_session rank -1 2024-03-29T11:19:32.028 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T11:19:32.044+0000 7f1fa1ffb640 1 -- 172.21.15.84:0/3246226570 >> v1:172.21.15.84:6790/0 conn(0x7f1f840036e0 legacy=0x7f1f84005b30 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-29T11:19:32.028 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T11:19:32.044+0000 7f1fa1ffb640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:19:32.029 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T11:19:32.045+0000 7f1fa1ffb640 10 monclient(hunting): picked mon.a con 0x7f1f840072d0 addr v1:172.21.15.84:6789/0 2024-03-29T11:19:32.029 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T11:19:32.045+0000 7f1fa1ffb640 10 monclient(hunting): picked mon.c con 0x7f1f90007960 addr v1:172.21.15.84:6790/0 2024-03-29T11:19:32.029 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T11:19:32.045+0000 7f1fa1ffb640 10 monclient(hunting): picked mon.b con 0x7f1fa40b8240 addr v1:172.21.15.179:6789/0 2024-03-29T11:19:32.029 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T11:19:32.045+0000 7f1fa1ffb640 1 -- 172.21.15.84:0/3246226570 --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f1fa413fb60 con 0x7f1f840072d0 2024-03-29T11:19:32.029 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T11:19:32.045+0000 7f1fa1ffb640 1 -- 172.21.15.84:0/3246226570 --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f1f84010b10 con 0x7f1fa40b8240 2024-03-29T11:19:32.029 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T11:19:32.045+0000 7f1fa1ffb640 1 -- 172.21.15.84:0/3246226570 --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f1f90010ed0 con 0x7f1f90007960 2024-03-29T11:19:32.029 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T11:19:32.045+0000 7f1fa1ffb640 10 monclient(hunting): _renew_subs 2024-03-29T11:19:32.029 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T11:19:32.045+0000 7f1fa1ffb640 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.84:6790/0 2024-03-29T11:19:32.029 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T11:19:32.045+0000 7f1fa1ffb640 1 -- 172.21.15.84:0/3246226570 <== mon.0 v1:172.21.15.84:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1970784327 0 0) 0x7f1fa413fb60 con 0x7f1f840072d0 2024-03-29T11:19:32.029 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T11:19:32.045+0000 7f1fa1ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:19:32.029 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T11:19:32.045+0000 7f1fa1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:32.029 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T11:19:32.045+0000 7f1fa1ffb640 10 monclient(hunting): _init_auth already have auth, reseting 2024-03-29T11:19:32.029 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T11:19:32.045+0000 7f1fa37fe640 1 -- 172.21.15.84:0/3246226570 >> v1:172.21.15.179:6789/0 conn(0x7f1fa40b8240 legacy=0x7f1f9000d530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-03-29T11:19:32.030 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T11:19:32.045+0000 7f1fa37fe640 1 -- 172.21.15.84:0/3246226570 >> v1:172.21.15.179:6789/0 conn(0x7f1fa40b8240 legacy=0x7f1f9000d530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-03-29T11:19:32.030 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T11:19:32.045+0000 7f1fa37fe640 1 --1- 172.21.15.84:0/3246226570 >> v1:172.21.15.179:6789/0 conn(0x7f1fa40b8240 0x7f1f9000d530 :-1 s=CONNECTING_SEND_CONNECT_MSG pgs=0 cs=0 l=0).handle_connect_reply_1 read connect reply failed 2024-03-29T11:19:32.030 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T11:19:32.045+0000 7f1fa1ffb640 1 -- 172.21.15.84:0/3246226570 --> v1:172.21.15.84:6789/0 -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f1f90011060 con 0x7f1f840072d0 2024-03-29T11:19:32.030 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T11:19:32.045+0000 7f1fa1ffb640 1 -- 172.21.15.84:0/3246226570 <== mon.0 v1:172.21.15.84:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 233+0+0 (unknown 2933989472 0 0) 0x7f1f90011060 con 0x7f1f840072d0 2024-03-29T11:19:32.030 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T11:19:32.045+0000 7f1fa1ffb640 1 -- 172.21.15.84:0/3246226570 >> v1:172.21.15.84:6790/0 conn(0x7f1f90007960 legacy=0x7f1f90009d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:32.030 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T11:19:32.045+0000 7f1fa1ffb640 1 -- 172.21.15.84:0/3246226570 >> v1:172.21.15.179:6789/0 conn(0x7f1fa40b8240 legacy=0x7f1f9000d530 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:19:32.030 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T11:19:32.045+0000 7f1fa1ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:19:32.030 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T11:19:32.045+0000 7f1fa1ffb640 1 monclient: found mon.a 2024-03-29T11:19:32.030 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T11:19:32.045+0000 7f1fa1ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:19:32.030 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T11:19:32.045+0000 7f1fa1ffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.84:6789/0 2024-03-29T11:19:32.030 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T11:19:32.045+0000 7f1fa1ffb640 1 -- 172.21.15.84:0/3246226570 --> v1:172.21.15.84:6789/0 -- mon_subscribe({config=0+,monmap=2+}) v3 -- 0x7f1f900121f0 con 0x7f1f840072d0 2024-03-29T11:19:32.031 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T11:19:32.045+0000 7f1fa1ffb640 10 monclient: _finish_auth 0 2024-03-29T11:19:32.031 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T11:19:32.045+0000 7f1fa1ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:19:32.031 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T11:19:32.045+0000 7f1fa1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:19:32.031 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T11:19:32.045+0000 7f1fa1ffb640 1 -- 172.21.15.84:0/3246226570 <== mon.0 v1:172.21.15.84:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4236868694 0 0) 0x7f1f94001d50 con 0x7f1f840072d0 2024-03-29T11:19:32.031 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T11:19:32.045+0000 7f1fa1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:32.031 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T11:19:32.045+0000 7f1fa1ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:19:32.031 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T11:19:32.045+0000 7f1fa1ffb640 10 monclient: dump: 2024-03-29T11:19:32.031 INFO:teuthology.orchestra.run.smithi084.stderr:epoch 1 2024-03-29T11:19:32.031 INFO:teuthology.orchestra.run.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:19:32.031 INFO:teuthology.orchestra.run.smithi084.stderr:last_changed 2024-03-29T11:18:53.864025+0000 2024-03-29T11:19:32.031 INFO:teuthology.orchestra.run.smithi084.stderr:created 2024-03-29T11:18:53.864025+0000 2024-03-29T11:19:32.031 INFO:teuthology.orchestra.run.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:32.032 INFO:teuthology.orchestra.run.smithi084.stderr:election_strategy: 1 2024-03-29T11:19:32.032 INFO:teuthology.orchestra.run.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T11:19:32.032 INFO:teuthology.orchestra.run.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T11:19:32.032 INFO:teuthology.orchestra.run.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T11:19:32.032 INFO:teuthology.orchestra.run.smithi084.stderr: 2024-03-29T11:19:32.032 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T11:19:32.045+0000 7f1fa99c6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:19:32.032 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T11:19:32.045+0000 7f1fa1ffb640 1 -- 172.21.15.84:0/3246226570 <== mon.0 v1:172.21.15.84:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1f94004770 con 0x7f1f840072d0 2024-03-29T11:19:32.032 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T11:19:32.045+0000 7f1fa1ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:19:32.032 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T11:19:32.045+0000 7f1fa99c6640 10 monclient: get_monmap_and_config success 2024-03-29T11:19:32.032 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T11:19:32.045+0000 7f1fa99c6640 4 set_mon_vals no callback set 2024-03-29T11:19:32.032 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T11:19:32.046+0000 7f1fa99c6640 10 monclient: shutdown 2024-03-29T11:19:32.032 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T11:19:32.046+0000 7f1fa99c6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:19:32.032 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T11:19:32.046+0000 7f1fa99c6640 1 -- 172.21.15.84:0/3246226570 >> v1:172.21.15.84:6789/0 conn(0x7f1f840072d0 legacy=0x7f1f90003cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:32.033 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T11:19:32.046+0000 7f1fa99c6640 1 -- 172.21.15.84:0/3246226570 shutdown_connections 2024-03-29T11:19:32.033 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T11:19:32.046+0000 7f1fa99c6640 1 -- 172.21.15.84:0/3246226570 >> 172.21.15.84:0/3246226570 conn(0x7f1fa4066b00 msgr2=0x7f1fa4066f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:19:32.033 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T11:19:32.047+0000 7f1fa99c6640 1 -- 172.21.15.84:0/3246226570 shutdown_connections 2024-03-29T11:19:32.033 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T11:19:32.047+0000 7f1fa99c6640 1 -- 172.21.15.84:0/3246226570 wait complete. 2024-03-29T11:19:32.033 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T11:19:32.047+0000 7f1fa99c6640 10 monclient: build_initial_monmap 2024-03-29T11:19:32.033 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T11:19:32.048+0000 7f1fa99c6640 10 monclient: monmap: 2024-03-29T11:19:32.033 INFO:teuthology.orchestra.run.smithi084.stderr:epoch 0 2024-03-29T11:19:32.033 INFO:teuthology.orchestra.run.smithi084.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:19:32.033 INFO:teuthology.orchestra.run.smithi084.stderr:last_changed 0.000000 2024-03-29T11:19:32.033 INFO:teuthology.orchestra.run.smithi084.stderr:created 0.000000 2024-03-29T11:19:32.033 INFO:teuthology.orchestra.run.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-29T11:19:32.033 INFO:teuthology.orchestra.run.smithi084.stderr:election_strategy: 1 2024-03-29T11:19:32.033 INFO:teuthology.orchestra.run.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-29T11:19:32.034 INFO:teuthology.orchestra.run.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.noname-b 2024-03-29T11:19:32.034 INFO:teuthology.orchestra.run.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-c 2024-03-29T11:19:32.034 INFO:teuthology.orchestra.run.smithi084.stderr: 2024-03-29T11:19:32.034 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T11:19:32.048+0000 7f1fa99c6640 1 librados: starting msgr at 2024-03-29T11:19:32.034 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T11:19:32.048+0000 7f1fa99c6640 1 librados: starting objecter 2024-03-29T11:19:32.034 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T11:19:32.048+0000 7f1fa99c6640 1 Processor -- start 2024-03-29T11:19:32.038 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T11:19:32.048+0000 7f1fa99c6640 1 -- start start 2024-03-29T11:19:32.038 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T11:19:32.048+0000 7f1fa99c6640 1 librados: setting wanted keys 2024-03-29T11:19:32.039 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T11:19:32.048+0000 7f1fa99c6640 1 librados: calling monclient init 2024-03-29T11:19:32.039 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T11:19:32.048+0000 7f1fa99c6640 10 monclient: init 2024-03-29T11:19:32.039 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T11:19:32.049+0000 7f1fa99c6640 10 monclient: _reopen_session rank -1 2024-03-29T11:19:32.039 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T11:19:32.049+0000 7f1fa99c6640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:19:32.039 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T11:19:32.049+0000 7f1fa99c6640 10 monclient(hunting): picked mon.noname-c con 0x7f1f840072d0 addr v1:172.21.15.84:6790/0 2024-03-29T11:19:32.039 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T11:19:32.049+0000 7f1fa99c6640 10 monclient(hunting): picked mon.noname-a con 0x7f1f90007960 addr v1:172.21.15.84:6789/0 2024-03-29T11:19:32.039 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T11:19:32.049+0000 7f1fa99c6640 10 monclient(hunting): picked mon.noname-b con 0x7f1fa40b8240 addr v1:172.21.15.179:6789/0 2024-03-29T11:19:32.039 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T11:19:32.049+0000 7f1fa99c6640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1fa41d80c0 con 0x7f1f90007960 2024-03-29T11:19:32.039 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T11:19:32.049+0000 7f1fa99c6640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1fa41d7d50 con 0x7f1fa40b8240 2024-03-29T11:19:32.039 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T11:19:32.049+0000 7f1fa99c6640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1fa41399b0 con 0x7f1f840072d0 2024-03-29T11:19:32.039 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T11:19:32.049+0000 7f1fa99c6640 10 monclient(hunting): _renew_subs 2024-03-29T11:19:32.039 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T11:19:32.049+0000 7f1fa99c6640 10 monclient(hunting): authenticate will time out at 833.343782s 2024-03-29T11:19:32.040 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T11:19:32.049+0000 7f1fa2ffd640 1 --1- >> v1:172.21.15.84:6790/0 conn(0x7f1f840072d0 0x7f1fa41d7640 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.84:6790/0 says I am v1:172.21.15.84:36022/0 (socket says 172.21.15.84:36022) 2024-03-29T11:19:32.040 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T11:19:32.049+0000 7f1fa27fc640 1 --1- >> v1:172.21.15.84:6789/0 conn(0x7f1f90007960 0x7f1fa40af870 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.84:6789/0 says I am v1:172.21.15.84:39016/0 (socket says 172.21.15.84:39016) 2024-03-29T11:19:32.040 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T11:19:32.049+0000 7f1fa2ffd640 1 -- 172.21.15.84:0/67149761 learned_addr learned my addr 172.21.15.84:0/67149761 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-29T11:19:32.040 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T11:19:32.049+0000 7f1f9bfff640 10 client.?.objecter ms_handle_connect 0x7f1f90007960 2024-03-29T11:19:32.040 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T11:19:32.049+0000 7f1f9bfff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:19:32.040 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T11:19:32.049+0000 7f1f9bfff640 10 client.?.objecter ms_handle_connect 0x7f1f840072d0 2024-03-29T11:19:32.040 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T11:19:32.049+0000 7f1f9bfff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:19:32.040 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T11:19:32.049+0000 7f1f9bfff640 10 client.?.objecter ms_handle_connect 0x7f1fa40b8240 2024-03-29T11:19:32.040 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T11:19:32.049+0000 7f1f9bfff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:19:32.040 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T11:19:32.050+0000 7f1f9bfff640 1 -- 172.21.15.84:0/67149761 <== mon.0 v1:172.21.15.84:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 927917572 0 0) 0x7f1fa41d80c0 con 0x7f1f90007960 2024-03-29T11:19:32.040 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T11:19:32.050+0000 7f1f9bfff640 10 monclient(hunting): _negotiate 2024-03-29T11:19:32.040 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T11:19:32.050+0000 7f1f9bfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:32.041 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T11:19:32.050+0000 7f1f9bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:32.041 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T11:19:32.050+0000 7f1f9bfff640 10 monclient(hunting): my global_id is 4113 2024-03-29T11:19:32.041 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T11:19:32.050+0000 7f1f9bfff640 1 -- 172.21.15.84:0/67149761 --> v1:172.21.15.84:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1f84010340 con 0x7f1f90007960 2024-03-29T11:19:32.041 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T11:19:32.050+0000 7f1f9bfff640 1 -- 172.21.15.84:0/67149761 <== mon.0 v1:172.21.15.84:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4203661208 0 0) 0x7f1f84010340 con 0x7f1f90007960 2024-03-29T11:19:32.041 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T11:19:32.050+0000 7f1f9bfff640 1 -- 172.21.15.84:0/67149761 --> v1:172.21.15.84:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1fa41d80c0 con 0x7f1f90007960 2024-03-29T11:19:32.041 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T11:19:32.050+0000 7f1f9bfff640 1 -- 172.21.15.84:0/67149761 <== mon.0 v1:172.21.15.84:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4236868694 0 0) 0x7f1f8c001930 con 0x7f1f90007960 2024-03-29T11:19:32.041 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T11:19:32.050+0000 7f1f9bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:32.041 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T11:19:32.050+0000 7f1f9bfff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:19:32.041 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T11:19:32.050+0000 7f1f9bfff640 10 monclient(hunting): dump: 2024-03-29T11:19:32.041 INFO:teuthology.orchestra.run.smithi084.stderr:epoch 1 2024-03-29T11:19:32.041 INFO:teuthology.orchestra.run.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:19:32.041 INFO:teuthology.orchestra.run.smithi084.stderr:last_changed 2024-03-29T11:18:53.864025+0000 2024-03-29T11:19:32.041 INFO:teuthology.orchestra.run.smithi084.stderr:created 2024-03-29T11:18:53.864025+0000 2024-03-29T11:19:32.042 INFO:teuthology.orchestra.run.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:32.042 INFO:teuthology.orchestra.run.smithi084.stderr:election_strategy: 1 2024-03-29T11:19:32.042 INFO:teuthology.orchestra.run.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T11:19:32.042 INFO:teuthology.orchestra.run.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T11:19:32.042 INFO:teuthology.orchestra.run.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T11:19:32.042 INFO:teuthology.orchestra.run.smithi084.stderr: 2024-03-29T11:19:32.042 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T11:19:32.050+0000 7f1f9bfff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:19:32.042 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T11:19:32.050+0000 7f1f9bfff640 1 -- 172.21.15.84:0/67149761 <== mon.0 v1:172.21.15.84:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3137055743 0 0) 0x7f1fa41d80c0 con 0x7f1f90007960 2024-03-29T11:19:32.042 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T11:19:32.050+0000 7f1f9bfff640 1 -- 172.21.15.84:0/67149761 >> v1:172.21.15.84:6790/0 conn(0x7f1f840072d0 legacy=0x7f1fa41d7640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:32.042 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T11:19:32.050+0000 7f1f9bfff640 1 -- 172.21.15.84:0/67149761 >> v1:172.21.15.179:6789/0 conn(0x7f1fa40b8240 legacy=0x7f1fa40aff80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:32.042 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T11:19:32.050+0000 7f1f9bfff640 10 monclient: _finish_hunting 0 2024-03-29T11:19:32.042 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T11:19:32.050+0000 7f1f9bfff640 1 monclient: found mon.a 2024-03-29T11:19:32.043 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T11:19:32.050+0000 7f1f9bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:19:32.043 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T11:19:32.050+0000 7f1f9bfff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.84:6789/0 2024-03-29T11:19:32.043 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T11:19:32.050+0000 7f1f9bfff640 1 -- 172.21.15.84:0/67149761 --> v1:172.21.15.84:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1fa4138f10 con 0x7f1f90007960 2024-03-29T11:19:32.043 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T11:19:32.050+0000 7f1f9bfff640 10 monclient: _finish_auth 0 2024-03-29T11:19:32.043 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T11:19:32.050+0000 7f1f9bfff640 10 monclient: _check_auth_tickets 2024-03-29T11:19:32.043 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T11:19:32.050+0000 7f1f9bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:19:32.043 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T11:19:32.050+0000 7f1fa99c6640 5 monclient: authenticate success, global_id 4113 2024-03-29T11:19:32.043 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T11:19:32.050+0000 7f1fa99c6640 10 monclient: _renew_subs 2024-03-29T11:19:32.043 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T11:19:32.050+0000 7f1fa99c6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.84:6789/0 2024-03-29T11:19:32.043 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T11:19:32.050+0000 7f1fa99c6640 1 -- 172.21.15.84:0/67149761 --> v1:172.21.15.84:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1fa4139180 con 0x7f1f90007960 2024-03-29T11:19:32.044 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T11:19:32.050+0000 7f1fa99c6640 10 client.4113.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:19:32.044 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T11:19:32.050+0000 7f1fa99c6640 10 monclient: _renew_subs 2024-03-29T11:19:32.044 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T11:19:32.050+0000 7f1f9bfff640 1 -- 172.21.15.84:0/67149761 <== mon.0 v1:172.21.15.84:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1f8c001d60 con 0x7f1f90007960 2024-03-29T11:19:32.044 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T11:19:32.050+0000 7f1fa99c6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.84:6789/0 2024-03-29T11:19:32.044 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T11:19:32.050+0000 7f1fa99c6640 1 -- 172.21.15.84:0/67149761 --> v1:172.21.15.84:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f1fa41e30b0 con 0x7f1f90007960 2024-03-29T11:19:32.044 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T11:19:32.050+0000 7f1fa99c6640 1 librados: init done 2024-03-29T11:19:32.044 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T11:19:32.050+0000 7f1f9bfff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:19:32.044 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T11:19:32.050+0000 7f1f9bfff640 1 -- 172.21.15.84:0/67149761 <== mon.0 v1:172.21.15.84:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4236868694 0 0) 0x7f1f8c004ac0 con 0x7f1f90007960 2024-03-29T11:19:32.044 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T11:19:32.050+0000 7f1f9bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:32.044 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T11:19:32.050+0000 7f1f9bfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:19:32.044 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T11:19:32.050+0000 7f1f9bfff640 10 monclient: dump: 2024-03-29T11:19:32.044 INFO:teuthology.orchestra.run.smithi084.stderr:epoch 1 2024-03-29T11:19:32.044 INFO:teuthology.orchestra.run.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:19:32.045 INFO:teuthology.orchestra.run.smithi084.stderr:last_changed 2024-03-29T11:18:53.864025+0000 2024-03-29T11:19:32.045 INFO:teuthology.orchestra.run.smithi084.stderr:created 2024-03-29T11:18:53.864025+0000 2024-03-29T11:19:32.045 INFO:teuthology.orchestra.run.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:32.045 INFO:teuthology.orchestra.run.smithi084.stderr:election_strategy: 1 2024-03-29T11:19:32.045 INFO:teuthology.orchestra.run.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T11:19:32.045 INFO:teuthology.orchestra.run.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T11:19:32.045 INFO:teuthology.orchestra.run.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T11:19:32.045 INFO:teuthology.orchestra.run.smithi084.stderr: 2024-03-29T11:19:32.045 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T11:19:32.050+0000 7f1fa17fa640 4 set_mon_vals no callback set 2024-03-29T11:19:32.045 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T11:19:32.050+0000 7f1f9bfff640 1 -- 172.21.15.84:0/67149761 <== mon.0 v1:172.21.15.84:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f1f8c005030 con 0x7f1f90007960 2024-03-29T11:19:32.045 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T11:19:32.051+0000 7f1f9bfff640 1 -- 172.21.15.84:0/67149761 <== mon.0 v1:172.21.15.84:6789/0 8 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (unknown 3354100349 0 0) 0x7f1f8c006690 con 0x7f1f90007960 2024-03-29T11:19:32.045 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T11:19:32.051+0000 7f1f9bfff640 10 client.4113.objecter ms_dispatch 0x7f1fa4000b90 osd_map(1..1 src has 1..1) v4 2024-03-29T11:19:32.045 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T11:19:32.051+0000 7f1f9bfff640 3 client.4113.objecter handle_osd_map got epochs [1,1] > 0 2024-03-29T11:19:32.046 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T11:19:32.051+0000 7f1f9bfff640 3 client.4113.objecter handle_osd_map decoding full epoch 1 2024-03-29T11:19:32.046 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T11:19:32.051+0000 7f1f9bfff640 20 client.4113.objecter dump_active .. 0 homeless 2024-03-29T11:19:32.046 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T11:19:32.052+0000 7f1fa99c6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:32.046 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T11:19:32.052+0000 7f1fa99c6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:32.046 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T11:19:32.052+0000 7f1fa99c6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.84:6789/0 2024-03-29T11:19:32.046 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T11:19:32.052+0000 7f1fa99c6640 1 -- 172.21.15.84:0/67149761 --> v1:172.21.15.84:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1fa409b170 con 0x7f1f90007960 2024-03-29T11:19:32.046 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T11:19:32.055+0000 7f1f9bfff640 1 -- 172.21.15.84:0/67149761 <== mon.0 v1:172.21.15.84:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f1f8c0052e0 con 0x7f1f90007960 2024-03-29T11:19:32.046 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T11:19:32.055+0000 7f1f9bfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:32.046 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T11:19:32.055+0000 7f1f9bfff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:19:32.101 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T11:19:32.124+0000 7f1fa99c6640 10 start_mon_command cmd=[{"prefix": "osd crush tunables", "profile": "default"}] 2024-03-29T11:19:32.101 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T11:19:32.124+0000 7f1fa99c6640 10 monclient: _send_command 2 [{"prefix": "osd crush tunables", "profile": "default"}] 2024-03-29T11:19:32.101 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T11:19:32.124+0000 7f1fa99c6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.84:6789/0 2024-03-29T11:19:32.101 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T11:19:32.124+0000 7f1fa99c6640 1 -- 172.21.15.84:0/67149761 --> v1:172.21.15.84:6789/0 -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) v1 -- 0x7f1fa404f730 con 0x7f1f90007960 2024-03-29T11:19:32.102 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T11:19:32.126+0000 7f1f9bfff640 1 -- 172.21.15.84:0/67149761 <== mon.0 v1:172.21.15.84:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v2) v1 ==== 124+0+0 (unknown 2277712955 0 0) 0x7f1f8c005660 con 0x7f1f90007960 2024-03-29T11:19:32.103 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T11:19:32.126+0000 7f1f9bfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush tunables", "profile": "default"}] 2024-03-29T11:19:32.103 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T11:19:32.126+0000 7f1f9bfff640 10 monclient: _finish_command 2 = system:0 adjusted tunables profile to default 2024-03-29T11:19:32.103 INFO:teuthology.orchestra.run.smithi084.stderr:adjusted tunables profile to default 2024-03-29T11:19:32.104 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T11:19:32.127+0000 7f1fa99c6640 10 librados: watch_flush enter 2024-03-29T11:19:32.104 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T11:19:32.127+0000 7f1fa99c6640 10 librados: watch_flush exit 2024-03-29T11:19:32.104 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T11:19:32.127+0000 7f1fa99c6640 20 client.4113.objecter shutdown clearing up homeless session... 2024-03-29T11:19:32.104 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T11:19:32.127+0000 7f1fa99c6640 10 client.4113.objecter successfully canceled tick 2024-03-29T11:19:32.104 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T11:19:32.127+0000 7f1fa99c6640 10 monclient: shutdown 2024-03-29T11:19:32.105 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T11:19:32.127+0000 7f1fa99c6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:19:32.105 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T11:19:32.127+0000 7f1fa99c6640 1 -- 172.21.15.84:0/67149761 >> v1:172.21.15.84:6789/0 conn(0x7f1f90007960 legacy=0x7f1fa40af870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:32.105 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T11:19:32.127+0000 7f1fa99c6640 1 -- 172.21.15.84:0/67149761 shutdown_connections 2024-03-29T11:19:32.105 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T11:19:32.127+0000 7f1fa99c6640 1 -- 172.21.15.84:0/67149761 >> 172.21.15.84:0/67149761 conn(0x7f1fa4066b00 msgr2=0x7f1fa406b230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:19:32.105 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T11:19:32.127+0000 7f1fa99c6640 1 -- 172.21.15.84:0/67149761 shutdown_connections 2024-03-29T11:19:32.105 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T11:19:32.127+0000 7f1fa99c6640 1 -- 172.21.15.84:0/67149761 wait complete. 2024-03-29T11:19:32.105 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T11:19:32.127+0000 7f1fa99c6640 1 librados: shutdown 2024-03-29T11:19:32.115 INFO:tasks.ceph:check_enable_crimson: False 2024-03-29T11:19:32.115 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2024-03-29T11:19:32.115 DEBUG:teuthology.orchestra.run.smithi084:> set -ex 2024-03-29T11:19:32.115 DEBUG:teuthology.orchestra.run.smithi084:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2024-03-29T11:19:32.143 DEBUG:teuthology.orchestra.run.smithi084:> set -ex 2024-03-29T11:19:32.143 DEBUG:teuthology.orchestra.run.smithi084:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2024-03-29T11:19:32.210 DEBUG:teuthology.orchestra.run.smithi084:> set -ex 2024-03-29T11:19:32.210 DEBUG:teuthology.orchestra.run.smithi084:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2024-03-29T11:19:32.278 DEBUG:teuthology.orchestra.run.smithi084:> set -ex 2024-03-29T11:19:32.278 DEBUG:teuthology.orchestra.run.smithi084:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2024-03-29T11:19:32.345 DEBUG:teuthology.orchestra.run.smithi179:> set -ex 2024-03-29T11:19:32.345 DEBUG:teuthology.orchestra.run.smithi179:> sudo dd if=/var/lib/ceph/osd/ceph-4/fsid of=/dev/stdout 2024-03-29T11:19:32.373 DEBUG:teuthology.orchestra.run.smithi179:> set -ex 2024-03-29T11:19:32.373 DEBUG:teuthology.orchestra.run.smithi179:> sudo dd if=/var/lib/ceph/osd/ceph-5/fsid of=/dev/stdout 2024-03-29T11:19:32.440 DEBUG:teuthology.orchestra.run.smithi179:> set -ex 2024-03-29T11:19:32.440 DEBUG:teuthology.orchestra.run.smithi179:> sudo dd if=/var/lib/ceph/osd/ceph-6/fsid of=/dev/stdout 2024-03-29T11:19:32.507 DEBUG:teuthology.orchestra.run.smithi179:> set -ex 2024-03-29T11:19:32.507 DEBUG:teuthology.orchestra.run.smithi179:> sudo dd if=/var/lib/ceph/osd/ceph-7/fsid of=/dev/stdout 2024-03-29T11:19:32.520 INFO:tasks.ceph.mgr.x.smithi179.stderr:2024-03-29T11:19:32.543+0000 7f8706237040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-29T11:19:32.527 INFO:tasks.ceph.mgr.y.smithi084.stderr:2024-03-29T11:19:32.551+0000 7f45957a8040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-29T11:19:32.545 DEBUG:teuthology.orchestra.run.smithi179:> sudo ceph --cluster ceph osd new 5c39e276-2034-4716-bbc0-57a634eefa92 0 2024-03-29T11:19:32.576 INFO:tasks.ceph.mgr.x.smithi179.stderr:2024-03-29T11:19:32.600+0000 7f8706237040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-29T11:19:32.581 INFO:tasks.ceph.mgr.y.smithi084.stderr:2024-03-29T11:19:32.605+0000 7f45957a8040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-29T11:19:32.676 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:32.699+0000 7f7f27baf640 10 monclient: get_monmap_and_config 2024-03-29T11:19:32.676 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:32.699+0000 7f7f27baf640 10 monclient: build_initial_monmap 2024-03-29T11:19:32.677 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:32.699+0000 7f7f27baf640 10 monclient: monmap: 2024-03-29T11:19:32.677 INFO:teuthology.orchestra.run.smithi179.stderr:epoch 0 2024-03-29T11:19:32.677 INFO:teuthology.orchestra.run.smithi179.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:19:32.677 INFO:teuthology.orchestra.run.smithi179.stderr:last_changed 2024-03-29T11:19:32.700386+0000 2024-03-29T11:19:32.677 INFO:teuthology.orchestra.run.smithi179.stderr:created 2024-03-29T11:19:32.700386+0000 2024-03-29T11:19:32.677 INFO:teuthology.orchestra.run.smithi179.stderr:min_mon_release 0 (unknown) 2024-03-29T11:19:32.677 INFO:teuthology.orchestra.run.smithi179.stderr:election_strategy: 1 2024-03-29T11:19:32.677 INFO:teuthology.orchestra.run.smithi179.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-29T11:19:32.677 INFO:teuthology.orchestra.run.smithi179.stderr:1: v1:172.21.15.179:6789/0 mon.noname-c 2024-03-29T11:19:32.677 INFO:teuthology.orchestra.run.smithi179.stderr:2: v1:172.21.15.84:6790/0 mon.noname-b 2024-03-29T11:19:32.677 INFO:teuthology.orchestra.run.smithi179.stderr: 2024-03-29T11:19:32.677 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:32.700+0000 7f7f27baf640 1 Processor -- start 2024-03-29T11:19:32.677 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:32.700+0000 7f7f27baf640 1 -- start start 2024-03-29T11:19:32.677 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:32.700+0000 7f7f27baf640 10 monclient: init 2024-03-29T11:19:32.688 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:32.700+0000 7f7f27baf640 10 monclient: _reopen_session rank -1 2024-03-29T11:19:32.688 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:32.700+0000 7f7f27baf640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:19:32.688 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:32.700+0000 7f7f27baf640 10 monclient(hunting): picked mon.noname-a con 0x7f7f200a9cc0 addr v1:172.21.15.84:6789/0 2024-03-29T11:19:32.688 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:32.700+0000 7f7f27baf640 10 monclient(hunting): picked mon.noname-b con 0x7f7f200aae80 addr v1:172.21.15.84:6790/0 2024-03-29T11:19:32.688 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:32.700+0000 7f7f27baf640 10 monclient(hunting): picked mon.noname-c con 0x7f7f200aa0a0 addr v1:172.21.15.179:6789/0 2024-03-29T11:19:32.688 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:32.700+0000 7f7f27baf640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7f200abc30 con 0x7f7f200a9cc0 2024-03-29T11:19:32.688 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:32.700+0000 7f7f27baf640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7f200abf10 con 0x7f7f200aa0a0 2024-03-29T11:19:32.688 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:32.700+0000 7f7f27baf640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7f200aa960 con 0x7f7f200aae80 2024-03-29T11:19:32.689 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:32.700+0000 7f7f27baf640 10 monclient(hunting): _renew_subs 2024-03-29T11:19:32.689 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:32.700+0000 7f7f27baf640 10 monclient(hunting): authenticate will time out at 811.033001s 2024-03-29T11:19:32.689 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:32.700+0000 7f7f2614c640 1 --1- >> v1:172.21.15.179:6789/0 conn(0x7f7f200aa0a0 0x7f7f20149db0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.179:6789/0 says I am v1:172.21.15.179:53384/0 (socket says 172.21.15.179:53384) 2024-03-29T11:19:32.689 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:32.701+0000 7f7f2614c640 1 -- 172.21.15.179:0/509334421 learned_addr learned my addr 172.21.15.179:0/509334421 (peer_addr_for_me v1:172.21.15.179:0/0) 2024-03-29T11:19:32.689 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:32.701+0000 7f7f24949640 1 -- 172.21.15.179:0/509334421 <== mon.1 v1:172.21.15.179:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 487129090 0 0) 0x7f7f200abf10 con 0x7f7f200aa0a0 2024-03-29T11:19:32.689 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:32.701+0000 7f7f24949640 10 monclient(hunting): _negotiate 2024-03-29T11:19:32.689 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:32.701+0000 7f7f24949640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:32.689 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:32.701+0000 7f7f24949640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:32.689 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:32.701+0000 7f7f24949640 10 monclient(hunting): my global_id is 4099 2024-03-29T11:19:32.689 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:32.701+0000 7f7f24949640 1 -- 172.21.15.179:0/509334421 --> v1:172.21.15.179:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7f0c004630 con 0x7f7f200aa0a0 2024-03-29T11:19:32.689 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:32.701+0000 7f7f24949640 1 -- 172.21.15.179:0/509334421 <== mon.0 v1:172.21.15.84:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 217193106 0 0) 0x7f7f200abc30 con 0x7f7f200a9cc0 2024-03-29T11:19:32.689 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:32.701+0000 7f7f24949640 10 monclient(hunting): _negotiate 2024-03-29T11:19:32.689 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:32.701+0000 7f7f24949640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:32.689 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:32.701+0000 7f7f24949640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:32.689 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:32.701+0000 7f7f24949640 10 monclient(hunting): my global_id is 4116 2024-03-29T11:19:32.690 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:32.701+0000 7f7f24949640 1 -- 172.21.15.179:0/509334421 --> v1:172.21.15.84:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7f200abf10 con 0x7f7f200a9cc0 2024-03-29T11:19:32.690 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:32.701+0000 7f7f24949640 1 -- 172.21.15.179:0/509334421 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1992356364 0 0) 0x7f7f200aa960 con 0x7f7f200aae80 2024-03-29T11:19:32.690 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:32.701+0000 7f7f24949640 10 monclient(hunting): _negotiate 2024-03-29T11:19:32.690 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:32.701+0000 7f7f24949640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:32.690 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:32.701+0000 7f7f24949640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:32.690 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:32.701+0000 7f7f24949640 10 monclient(hunting): my global_id is 4115 2024-03-29T11:19:32.690 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:32.701+0000 7f7f24949640 1 -- 172.21.15.179:0/509334421 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7f200abc30 con 0x7f7f200aae80 2024-03-29T11:19:32.690 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:32.701+0000 7f7f24949640 1 -- 172.21.15.179:0/509334421 <== mon.1 v1:172.21.15.179:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2767802673 0 0) 0x7f7f0c004630 con 0x7f7f200aa0a0 2024-03-29T11:19:32.690 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:32.701+0000 7f7f24949640 1 -- 172.21.15.179:0/509334421 >> v1:172.21.15.84:6790/0 conn(0x7f7f200aae80 legacy=0x7f7f2013e920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:32.690 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:32.701+0000 7f7f24949640 1 -- 172.21.15.179:0/509334421 >> v1:172.21.15.84:6789/0 conn(0x7f7f200a9cc0 legacy=0x7f7f200ac610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:32.690 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:32.701+0000 7f7f24949640 10 monclient: _finish_hunting 0 2024-03-29T11:19:32.690 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:32.701+0000 7f7f24949640 1 monclient: found mon.noname-c 2024-03-29T11:19:32.690 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:32.701+0000 7f7f24949640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:19:32.690 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:32.701+0000 7f7f24949640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.179:6789/0 2024-03-29T11:19:32.690 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:32.701+0000 7f7f24949640 1 -- 172.21.15.179:0/509334421 --> v1:172.21.15.179:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f200ada90 con 0x7f7f200aa0a0 2024-03-29T11:19:32.691 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:32.701+0000 7f7f24949640 10 monclient: _finish_auth 0 2024-03-29T11:19:32.691 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:32.701+0000 7f7f24949640 10 monclient: _check_auth_tickets 2024-03-29T11:19:32.691 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:32.701+0000 7f7f24949640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:19:32.691 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:32.701+0000 7f7f27baf640 5 monclient: authenticate success, global_id 4099 2024-03-29T11:19:32.691 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:32.701+0000 7f7f24949640 1 -- 172.21.15.179:0/509334421 <== mon.1 v1:172.21.15.179:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4236868694 0 0) 0x7f7f18001a10 con 0x7f7f200aa0a0 2024-03-29T11:19:32.691 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:32.701+0000 7f7f24949640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:32.691 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:32.701+0000 7f7f24949640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:19:32.691 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:32.701+0000 7f7f24949640 10 monclient: dump: 2024-03-29T11:19:32.691 INFO:teuthology.orchestra.run.smithi179.stderr:epoch 1 2024-03-29T11:19:32.691 INFO:teuthology.orchestra.run.smithi179.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:19:32.691 INFO:teuthology.orchestra.run.smithi179.stderr:last_changed 2024-03-29T11:18:53.864025+0000 2024-03-29T11:19:32.691 INFO:teuthology.orchestra.run.smithi179.stderr:created 2024-03-29T11:18:53.864025+0000 2024-03-29T11:19:32.691 INFO:teuthology.orchestra.run.smithi179.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:32.691 INFO:teuthology.orchestra.run.smithi179.stderr:election_strategy: 1 2024-03-29T11:19:32.691 INFO:teuthology.orchestra.run.smithi179.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T11:19:32.691 INFO:teuthology.orchestra.run.smithi179.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T11:19:32.692 INFO:teuthology.orchestra.run.smithi179.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T11:19:32.692 INFO:teuthology.orchestra.run.smithi179.stderr: 2024-03-29T11:19:32.692 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:32.701+0000 7f7f27baf640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:19:32.692 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:32.701+0000 7f7f24949640 1 -- 172.21.15.179:0/509334421 <== mon.1 v1:172.21.15.179:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7f18001bf0 con 0x7f7f200aa0a0 2024-03-29T11:19:32.692 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:32.701+0000 7f7f24949640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:19:32.692 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:32.701+0000 7f7f27baf640 10 monclient: get_monmap_and_config success 2024-03-29T11:19:32.692 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:32.701+0000 7f7f24949640 1 -- 172.21.15.179:0/509334421 <== mon.1 v1:172.21.15.179:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4236868694 0 0) 0x7f7f18002000 con 0x7f7f200aa0a0 2024-03-29T11:19:32.692 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:32.701+0000 7f7f27baf640 4 set_mon_vals no callback set 2024-03-29T11:19:32.692 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:32.701+0000 7f7f24949640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:32.692 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:32.701+0000 7f7f24949640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:19:32.692 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:32.701+0000 7f7f24949640 10 monclient: dump: 2024-03-29T11:19:32.692 INFO:teuthology.orchestra.run.smithi179.stderr:epoch 1 2024-03-29T11:19:32.692 INFO:teuthology.orchestra.run.smithi179.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:19:32.692 INFO:teuthology.orchestra.run.smithi179.stderr:last_changed 2024-03-29T11:18:53.864025+0000 2024-03-29T11:19:32.692 INFO:teuthology.orchestra.run.smithi179.stderr:created 2024-03-29T11:18:53.864025+0000 2024-03-29T11:19:32.692 INFO:teuthology.orchestra.run.smithi179.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:32.692 INFO:teuthology.orchestra.run.smithi179.stderr:election_strategy: 1 2024-03-29T11:19:32.693 INFO:teuthology.orchestra.run.smithi179.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T11:19:32.693 INFO:teuthology.orchestra.run.smithi179.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T11:19:32.693 INFO:teuthology.orchestra.run.smithi179.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T11:19:32.693 INFO:teuthology.orchestra.run.smithi179.stderr: 2024-03-29T11:19:32.693 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:32.702+0000 7f7f27baf640 10 monclient: shutdown 2024-03-29T11:19:32.693 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:32.702+0000 7f7f27baf640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:19:32.693 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:32.702+0000 7f7f27baf640 1 -- 172.21.15.179:0/509334421 >> v1:172.21.15.179:6789/0 conn(0x7f7f200aa0a0 legacy=0x7f7f20149db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:32.693 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:32.702+0000 7f7f27baf640 1 -- 172.21.15.179:0/509334421 shutdown_connections 2024-03-29T11:19:32.693 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:32.702+0000 7f7f27baf640 1 -- 172.21.15.179:0/509334421 >> 172.21.15.179:0/509334421 conn(0x7f7f200b3750 msgr2=0x7f7f200b3b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:19:32.693 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:32.702+0000 7f7f27baf640 1 -- 172.21.15.179:0/509334421 shutdown_connections 2024-03-29T11:19:32.693 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:32.702+0000 7f7f27baf640 1 -- 172.21.15.179:0/509334421 wait complete. 2024-03-29T11:19:32.693 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:32.702+0000 7f7f27baf640 10 monclient: build_initial_monmap 2024-03-29T11:19:32.693 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:32.702+0000 7f7f27baf640 10 monclient: monmap: 2024-03-29T11:19:32.693 INFO:teuthology.orchestra.run.smithi179.stderr:epoch 0 2024-03-29T11:19:32.693 INFO:teuthology.orchestra.run.smithi179.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:19:32.693 INFO:teuthology.orchestra.run.smithi179.stderr:last_changed 0.000000 2024-03-29T11:19:32.694 INFO:teuthology.orchestra.run.smithi179.stderr:created 0.000000 2024-03-29T11:19:32.694 INFO:teuthology.orchestra.run.smithi179.stderr:min_mon_release 0 (unknown) 2024-03-29T11:19:32.694 INFO:teuthology.orchestra.run.smithi179.stderr:election_strategy: 1 2024-03-29T11:19:32.694 INFO:teuthology.orchestra.run.smithi179.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-29T11:19:32.694 INFO:teuthology.orchestra.run.smithi179.stderr:1: v1:172.21.15.179:6789/0 mon.noname-b 2024-03-29T11:19:32.694 INFO:teuthology.orchestra.run.smithi179.stderr:2: v1:172.21.15.84:6790/0 mon.noname-c 2024-03-29T11:19:32.694 INFO:teuthology.orchestra.run.smithi179.stderr: 2024-03-29T11:19:32.694 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:32.703+0000 7f7f27baf640 1 librados: starting msgr at 2024-03-29T11:19:32.694 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:32.703+0000 7f7f27baf640 1 librados: starting objecter 2024-03-29T11:19:32.694 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:32.703+0000 7f7f27baf640 1 Processor -- start 2024-03-29T11:19:32.694 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:32.703+0000 7f7f27baf640 1 -- start start 2024-03-29T11:19:32.694 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:32.703+0000 7f7f27baf640 1 librados: setting wanted keys 2024-03-29T11:19:32.695 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:32.703+0000 7f7f27baf640 1 librados: calling monclient init 2024-03-29T11:19:32.695 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:32.703+0000 7f7f27baf640 10 monclient: init 2024-03-29T11:19:32.695 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:32.703+0000 7f7f27baf640 10 monclient: _reopen_session rank -1 2024-03-29T11:19:32.695 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:32.703+0000 7f7f27baf640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:19:32.695 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:32.703+0000 7f7f27baf640 10 monclient(hunting): picked mon.noname-a con 0x7f7f200a9cc0 addr v1:172.21.15.84:6789/0 2024-03-29T11:19:32.695 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:32.703+0000 7f7f27baf640 10 monclient(hunting): picked mon.noname-b con 0x7f7f200aa0a0 addr v1:172.21.15.179:6789/0 2024-03-29T11:19:32.695 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:32.703+0000 7f7f27baf640 10 monclient(hunting): picked mon.noname-c con 0x7f7f200aae80 addr v1:172.21.15.84:6790/0 2024-03-29T11:19:32.695 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:32.703+0000 7f7f27baf640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7f201d6970 con 0x7f7f200a9cc0 2024-03-29T11:19:32.695 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:32.703+0000 7f7f27baf640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7f2006dee0 con 0x7f7f200aa0a0 2024-03-29T11:19:32.695 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:32.703+0000 7f7f27baf640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7f201335e0 con 0x7f7f200aae80 2024-03-29T11:19:32.696 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:32.703+0000 7f7f27baf640 10 monclient(hunting): _renew_subs 2024-03-29T11:19:32.696 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:32.703+0000 7f7f27baf640 10 monclient(hunting): authenticate will time out at 811.035890s 2024-03-29T11:19:32.696 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:32.703+0000 7f7f2514a640 1 --1- >> v1:172.21.15.179:6789/0 conn(0x7f7f200aa0a0 0x7f7f20132a90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.179:6789/0 says I am v1:172.21.15.179:53392/0 (socket says 172.21.15.179:53392) 2024-03-29T11:19:32.696 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:32.703+0000 7f7f2594b640 1 --1- >> v1:172.21.15.84:6789/0 conn(0x7f7f200a9cc0 0x7f7f201d6260 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.84:6789/0 says I am v1:172.21.15.179:50946/0 (socket says 172.21.15.179:50946) 2024-03-29T11:19:32.696 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:32.703+0000 7f7f2514a640 1 -- 172.21.15.179:0/3808594583 learned_addr learned my addr 172.21.15.179:0/3808594583 (peer_addr_for_me v1:172.21.15.179:0/0) 2024-03-29T11:19:32.696 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:32.703+0000 7f7f127fc640 10 client.?.objecter ms_handle_connect 0x7f7f200a9cc0 2024-03-29T11:19:32.696 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:32.704+0000 7f7f127fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:19:32.696 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:32.704+0000 7f7f127fc640 10 client.?.objecter ms_handle_connect 0x7f7f200aa0a0 2024-03-29T11:19:32.696 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:32.704+0000 7f7f127fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:19:32.697 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:32.704+0000 7f7f127fc640 10 client.?.objecter ms_handle_connect 0x7f7f200aae80 2024-03-29T11:19:32.697 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:32.704+0000 7f7f127fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:19:32.697 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:32.704+0000 7f7f127fc640 1 -- 172.21.15.179:0/3808594583 <== mon.0 v1:172.21.15.84:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3330907103 0 0) 0x7f7f201d6970 con 0x7f7f200a9cc0 2024-03-29T11:19:32.697 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:32.704+0000 7f7f127fc640 10 monclient(hunting): _negotiate 2024-03-29T11:19:32.697 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:32.704+0000 7f7f127fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:32.697 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:32.704+0000 7f7f127fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:32.697 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:32.704+0000 7f7f127fc640 10 monclient(hunting): my global_id is 4119 2024-03-29T11:19:32.697 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:32.704+0000 7f7f127fc640 1 -- 172.21.15.179:0/3808594583 --> v1:172.21.15.84:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7f04004eb0 con 0x7f7f200a9cc0 2024-03-29T11:19:32.698 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:32.704+0000 7f7f127fc640 1 -- 172.21.15.179:0/3808594583 <== mon.1 v1:172.21.15.179:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4237571228 0 0) 0x7f7f2006dee0 con 0x7f7f200aa0a0 2024-03-29T11:19:32.698 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:32.704+0000 7f7f127fc640 10 monclient(hunting): _negotiate 2024-03-29T11:19:32.698 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:32.704+0000 7f7f127fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:32.698 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:32.704+0000 7f7f127fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:32.698 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:32.704+0000 7f7f127fc640 10 monclient(hunting): my global_id is 4102 2024-03-29T11:19:32.698 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:32.704+0000 7f7f127fc640 1 -- 172.21.15.179:0/3808594583 --> v1:172.21.15.179:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7f201d6970 con 0x7f7f200aa0a0 2024-03-29T11:19:32.698 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:32.704+0000 7f7f127fc640 1 -- 172.21.15.179:0/3808594583 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2363093487 0 0) 0x7f7f201335e0 con 0x7f7f200aae80 2024-03-29T11:19:32.698 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:32.704+0000 7f7f127fc640 10 monclient(hunting): _negotiate 2024-03-29T11:19:32.698 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:32.704+0000 7f7f127fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:32.699 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:32.704+0000 7f7f127fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:32.699 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:32.704+0000 7f7f127fc640 10 monclient(hunting): my global_id is 4118 2024-03-29T11:19:32.699 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:32.704+0000 7f7f127fc640 1 -- 172.21.15.179:0/3808594583 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7f2006dee0 con 0x7f7f200aae80 2024-03-29T11:19:32.699 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:32.704+0000 7f7f127fc640 1 -- 172.21.15.179:0/3808594583 <== mon.0 v1:172.21.15.84:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1840473501 0 0) 0x7f7f04004eb0 con 0x7f7f200a9cc0 2024-03-29T11:19:32.699 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:32.704+0000 7f7f127fc640 1 -- 172.21.15.179:0/3808594583 --> v1:172.21.15.84:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7f201335e0 con 0x7f7f200a9cc0 2024-03-29T11:19:32.699 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:32.704+0000 7f7f127fc640 1 -- 172.21.15.179:0/3808594583 <== mon.0 v1:172.21.15.84:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4236868694 0 0) 0x7f7f140019f0 con 0x7f7f200a9cc0 2024-03-29T11:19:32.699 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:32.704+0000 7f7f127fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:32.699 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:32.704+0000 7f7f127fc640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:19:32.699 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:32.704+0000 7f7f127fc640 10 monclient(hunting): dump: 2024-03-29T11:19:32.700 INFO:teuthology.orchestra.run.smithi179.stderr:epoch 1 2024-03-29T11:19:32.700 INFO:teuthology.orchestra.run.smithi179.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:19:32.700 INFO:teuthology.orchestra.run.smithi179.stderr:last_changed 2024-03-29T11:18:53.864025+0000 2024-03-29T11:19:32.700 INFO:teuthology.orchestra.run.smithi179.stderr:created 2024-03-29T11:18:53.864025+0000 2024-03-29T11:19:32.700 INFO:teuthology.orchestra.run.smithi179.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:32.700 INFO:teuthology.orchestra.run.smithi179.stderr:election_strategy: 1 2024-03-29T11:19:32.700 INFO:teuthology.orchestra.run.smithi179.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T11:19:32.700 INFO:teuthology.orchestra.run.smithi179.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T11:19:32.700 INFO:teuthology.orchestra.run.smithi179.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T11:19:32.700 INFO:teuthology.orchestra.run.smithi179.stderr: 2024-03-29T11:19:32.701 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:32.704+0000 7f7f127fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:19:32.701 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:32.704+0000 7f7f127fc640 1 -- 172.21.15.179:0/3808594583 <== mon.2 v1:172.21.15.84:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1848255711 0 0) 0x7f7f2006dee0 con 0x7f7f200aae80 2024-03-29T11:19:32.701 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:32.704+0000 7f7f127fc640 1 -- 172.21.15.179:0/3808594583 --> v1:172.21.15.84:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7f04004eb0 con 0x7f7f200aae80 2024-03-29T11:19:32.701 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:32.704+0000 7f7f127fc640 1 -- 172.21.15.179:0/3808594583 <== mon.2 v1:172.21.15.84:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4236868694 0 0) 0x7f7f18001d60 con 0x7f7f200aae80 2024-03-29T11:19:32.701 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:32.704+0000 7f7f127fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:32.701 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:32.704+0000 7f7f127fc640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:19:32.701 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:32.704+0000 7f7f127fc640 10 monclient(hunting): dump: 2024-03-29T11:19:32.701 INFO:teuthology.orchestra.run.smithi179.stderr:epoch 1 2024-03-29T11:19:32.701 INFO:teuthology.orchestra.run.smithi179.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:19:32.702 INFO:teuthology.orchestra.run.smithi179.stderr:last_changed 2024-03-29T11:18:53.864025+0000 2024-03-29T11:19:32.702 INFO:teuthology.orchestra.run.smithi179.stderr:created 2024-03-29T11:18:53.864025+0000 2024-03-29T11:19:32.702 INFO:teuthology.orchestra.run.smithi179.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:32.702 INFO:teuthology.orchestra.run.smithi179.stderr:election_strategy: 1 2024-03-29T11:19:32.702 INFO:teuthology.orchestra.run.smithi179.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T11:19:32.702 INFO:teuthology.orchestra.run.smithi179.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T11:19:32.703 INFO:teuthology.orchestra.run.smithi179.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T11:19:32.703 INFO:teuthology.orchestra.run.smithi179.stderr: 2024-03-29T11:19:32.703 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:32.704+0000 7f7f127fc640 1 -- 172.21.15.179:0/3808594583 <== mon.1 v1:172.21.15.179:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 247872482 0 0) 0x7f7f201d6970 con 0x7f7f200aa0a0 2024-03-29T11:19:32.703 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:32.704+0000 7f7f127fc640 1 -- 172.21.15.179:0/3808594583 --> v1:172.21.15.179:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7f2006dee0 con 0x7f7f200aa0a0 2024-03-29T11:19:32.703 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:32.704+0000 7f7f127fc640 1 -- 172.21.15.179:0/3808594583 <== mon.1 v1:172.21.15.179:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4236868694 0 0) 0x7f7f1c0017e0 con 0x7f7f200aa0a0 2024-03-29T11:19:32.703 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:32.704+0000 7f7f127fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:32.703 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:32.704+0000 7f7f127fc640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:19:32.703 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:32.704+0000 7f7f127fc640 10 monclient(hunting): dump: 2024-03-29T11:19:32.703 INFO:teuthology.orchestra.run.smithi179.stderr:epoch 1 2024-03-29T11:19:32.704 INFO:teuthology.orchestra.run.smithi179.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:19:32.704 INFO:teuthology.orchestra.run.smithi179.stderr:last_changed 2024-03-29T11:18:53.864025+0000 2024-03-29T11:19:32.704 INFO:teuthology.orchestra.run.smithi179.stderr:created 2024-03-29T11:18:53.864025+0000 2024-03-29T11:19:32.704 INFO:teuthology.orchestra.run.smithi179.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:32.704 INFO:teuthology.orchestra.run.smithi179.stderr:election_strategy: 1 2024-03-29T11:19:32.704 INFO:teuthology.orchestra.run.smithi179.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T11:19:32.704 INFO:teuthology.orchestra.run.smithi179.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T11:19:32.704 INFO:teuthology.orchestra.run.smithi179.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T11:19:32.704 INFO:teuthology.orchestra.run.smithi179.stderr: 2024-03-29T11:19:32.704 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:32.704+0000 7f7f127fc640 1 -- 172.21.15.179:0/3808594583 <== mon.0 v1:172.21.15.84:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4115490249 0 0) 0x7f7f201335e0 con 0x7f7f200a9cc0 2024-03-29T11:19:32.705 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:32.704+0000 7f7f127fc640 1 -- 172.21.15.179:0/3808594583 >> v1:172.21.15.84:6790/0 conn(0x7f7f200aae80 legacy=0x7f7f201de600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:32.705 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:32.704+0000 7f7f127fc640 1 -- 172.21.15.179:0/3808594583 >> v1:172.21.15.179:6789/0 conn(0x7f7f200aa0a0 legacy=0x7f7f20132a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:32.705 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:32.704+0000 7f7f127fc640 10 monclient: _finish_hunting 0 2024-03-29T11:19:32.705 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:32.704+0000 7f7f127fc640 1 monclient: found mon.a 2024-03-29T11:19:32.705 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:32.704+0000 7f7f127fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:19:32.705 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:32.704+0000 7f7f127fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.84:6789/0 2024-03-29T11:19:32.705 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:32.704+0000 7f7f127fc640 1 -- 172.21.15.179:0/3808594583 --> v1:172.21.15.84:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f20135500 con 0x7f7f200a9cc0 2024-03-29T11:19:32.706 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:32.704+0000 7f7f127fc640 10 monclient: _finish_auth 0 2024-03-29T11:19:32.706 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:32.704+0000 7f7f127fc640 10 monclient: _check_auth_tickets 2024-03-29T11:19:32.706 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:32.704+0000 7f7f127fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:19:32.706 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:32.704+0000 7f7f27baf640 5 monclient: authenticate success, global_id 4119 2024-03-29T11:19:32.706 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:32.704+0000 7f7f27baf640 10 monclient: _renew_subs 2024-03-29T11:19:32.706 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:32.704+0000 7f7f27baf640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.84:6789/0 2024-03-29T11:19:32.706 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:32.704+0000 7f7f27baf640 1 -- 172.21.15.179:0/3808594583 --> v1:172.21.15.84:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7f201331a0 con 0x7f7f200a9cc0 2024-03-29T11:19:32.707 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:32.704+0000 7f7f27baf640 10 client.4119.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:19:32.707 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:32.704+0000 7f7f27baf640 10 monclient: _renew_subs 2024-03-29T11:19:32.707 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:32.704+0000 7f7f27baf640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.84:6789/0 2024-03-29T11:19:32.707 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:32.704+0000 7f7f27baf640 1 -- 172.21.15.179:0/3808594583 --> v1:172.21.15.84:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f7f20134ee0 con 0x7f7f200a9cc0 2024-03-29T11:19:32.707 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:32.704+0000 7f7f27baf640 1 librados: init done 2024-03-29T11:19:32.707 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:32.704+0000 7f7f127fc640 1 -- 172.21.15.179:0/3808594583 <== mon.0 v1:172.21.15.84:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7f14001c90 con 0x7f7f200a9cc0 2024-03-29T11:19:32.707 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:32.704+0000 7f7f127fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:19:32.707 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:32.704+0000 7f7f127fc640 1 -- 172.21.15.179:0/3808594583 <== mon.0 v1:172.21.15.84:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4236868694 0 0) 0x7f7f14003a60 con 0x7f7f200a9cc0 2024-03-29T11:19:32.707 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:32.704+0000 7f7f127fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:32.708 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:32.704+0000 7f7f13fff640 4 set_mon_vals no callback set 2024-03-29T11:19:32.708 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:32.704+0000 7f7f127fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:19:32.708 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:32.704+0000 7f7f127fc640 10 monclient: dump: 2024-03-29T11:19:32.708 INFO:teuthology.orchestra.run.smithi179.stderr:epoch 1 2024-03-29T11:19:32.708 INFO:teuthology.orchestra.run.smithi179.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:19:32.708 INFO:teuthology.orchestra.run.smithi179.stderr:last_changed 2024-03-29T11:18:53.864025+0000 2024-03-29T11:19:32.708 INFO:teuthology.orchestra.run.smithi179.stderr:created 2024-03-29T11:18:53.864025+0000 2024-03-29T11:19:32.708 INFO:teuthology.orchestra.run.smithi179.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:32.708 INFO:teuthology.orchestra.run.smithi179.stderr:election_strategy: 1 2024-03-29T11:19:32.709 INFO:teuthology.orchestra.run.smithi179.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T11:19:32.709 INFO:teuthology.orchestra.run.smithi179.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T11:19:32.709 INFO:teuthology.orchestra.run.smithi179.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T11:19:32.709 INFO:teuthology.orchestra.run.smithi179.stderr: 2024-03-29T11:19:32.709 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:32.704+0000 7f7f127fc640 1 -- 172.21.15.179:0/3808594583 <== mon.0 v1:172.21.15.84:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f7f14004010 con 0x7f7f200a9cc0 2024-03-29T11:19:32.709 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:32.704+0000 7f7f127fc640 1 -- 172.21.15.179:0/3808594583 <== mon.0 v1:172.21.15.84:6789/0 8 ==== osd_map(2..2 src has 1..2) v4 ==== 725+0+0 (unknown 2790245867 0 0) 0x7f7f14006630 con 0x7f7f200a9cc0 2024-03-29T11:19:32.709 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:32.704+0000 7f7f127fc640 10 client.4119.objecter ms_dispatch 0x7f7f20000b90 osd_map(2..2 src has 1..2) v4 2024-03-29T11:19:32.709 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:32.704+0000 7f7f127fc640 3 client.4119.objecter handle_osd_map got epochs [2,2] > 0 2024-03-29T11:19:32.710 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:32.704+0000 7f7f127fc640 3 client.4119.objecter handle_osd_map decoding full epoch 2 2024-03-29T11:19:32.710 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:32.704+0000 7f7f127fc640 20 client.4119.objecter dump_active .. 0 homeless 2024-03-29T11:19:32.710 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:32.705+0000 7f7f27baf640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:32.710 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:32.705+0000 7f7f27baf640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:32.710 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:32.705+0000 7f7f27baf640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.84:6789/0 2024-03-29T11:19:32.710 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:32.705+0000 7f7f27baf640 1 -- 172.21.15.179:0/3808594583 --> v1:172.21.15.84:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7f08005080 con 0x7f7f200a9cc0 2024-03-29T11:19:32.710 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:32.707+0000 7f7f127fc640 1 -- 172.21.15.179:0/3808594583 <== mon.0 v1:172.21.15.84:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f7f14006930 con 0x7f7f200a9cc0 2024-03-29T11:19:32.710 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:32.707+0000 7f7f127fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:32.710 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:32.707+0000 7f7f127fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:19:32.756 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:32.779+0000 7f7f27baf640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "5c39e276-2034-4716-bbc0-57a634eefa92", "id": 0}] 2024-03-29T11:19:32.756 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:32.779+0000 7f7f27baf640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "5c39e276-2034-4716-bbc0-57a634eefa92", "id": 0}] 2024-03-29T11:19:32.756 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:32.779+0000 7f7f27baf640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.84:6789/0 2024-03-29T11:19:32.756 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:32.779+0000 7f7f27baf640 1 -- 172.21.15.179:0/3808594583 --> v1:172.21.15.84:6789/0 -- mon_command({"prefix": "osd new", "uuid": "5c39e276-2034-4716-bbc0-57a634eefa92", "id": 0} v 0) v1 -- 0x7f7f080058b0 con 0x7f7f200a9cc0 2024-03-29T11:19:32.758 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:32.781+0000 7f7f127fc640 1 -- 172.21.15.179:0/3808594583 <== mon.0 v1:172.21.15.84:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "5c39e276-2034-4716-bbc0-57a634eefa92", "id": 0}]=0 v3) v1 ==== 112+0+2 (unknown 3465501940 0 732707515) 0x7f7f14006b50 con 0x7f7f200a9cc0 2024-03-29T11:19:32.758 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:32.781+0000 7f7f127fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "5c39e276-2034-4716-bbc0-57a634eefa92", "id": 0}] 2024-03-29T11:19:32.758 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:32.781+0000 7f7f127fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:19:32.758 INFO:teuthology.orchestra.run.smithi179.stdout:0 2024-03-29T11:19:32.759 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:32.782+0000 7f7f27baf640 10 librados: watch_flush enter 2024-03-29T11:19:32.759 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:32.782+0000 7f7f27baf640 10 librados: watch_flush exit 2024-03-29T11:19:32.759 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:32.782+0000 7f7f27baf640 20 client.4119.objecter shutdown clearing up homeless session... 2024-03-29T11:19:32.759 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:32.782+0000 7f7f27baf640 10 client.4119.objecter successfully canceled tick 2024-03-29T11:19:32.759 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:32.782+0000 7f7f27baf640 10 monclient: shutdown 2024-03-29T11:19:32.759 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:32.782+0000 7f7f27baf640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:19:32.760 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:32.782+0000 7f7f27baf640 1 -- 172.21.15.179:0/3808594583 >> v1:172.21.15.84:6789/0 conn(0x7f7f200a9cc0 legacy=0x7f7f201d6260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:32.760 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:32.782+0000 7f7f27baf640 1 -- 172.21.15.179:0/3808594583 shutdown_connections 2024-03-29T11:19:32.760 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:32.782+0000 7f7f27baf640 1 -- 172.21.15.179:0/3808594583 >> 172.21.15.179:0/3808594583 conn(0x7f7f200b3750 msgr2=0x7f7f200b1340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:19:32.760 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:32.782+0000 7f7f27baf640 1 -- 172.21.15.179:0/3808594583 shutdown_connections 2024-03-29T11:19:32.760 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:32.782+0000 7f7f27baf640 1 -- 172.21.15.179:0/3808594583 wait complete. 2024-03-29T11:19:32.760 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:32.782+0000 7f7f27baf640 1 librados: shutdown 2024-03-29T11:19:32.769 DEBUG:teuthology.orchestra.run.smithi179:> sudo ceph --cluster ceph osd new cd580e99-109f-4bf1-88c0-a5987bc8b9ed 1 2024-03-29T11:19:32.868 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:32.891+0000 7fc4a134d640 10 monclient: get_monmap_and_config 2024-03-29T11:19:32.868 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:32.891+0000 7fc4a134d640 10 monclient: build_initial_monmap 2024-03-29T11:19:32.868 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:32.891+0000 7fc4a134d640 10 monclient: monmap: 2024-03-29T11:19:32.868 INFO:teuthology.orchestra.run.smithi179.stderr:epoch 0 2024-03-29T11:19:32.868 INFO:teuthology.orchestra.run.smithi179.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:19:32.868 INFO:teuthology.orchestra.run.smithi179.stderr:last_changed 2024-03-29T11:19:32.892660+0000 2024-03-29T11:19:32.868 INFO:teuthology.orchestra.run.smithi179.stderr:created 2024-03-29T11:19:32.892660+0000 2024-03-29T11:19:32.868 INFO:teuthology.orchestra.run.smithi179.stderr:min_mon_release 0 (unknown) 2024-03-29T11:19:32.868 INFO:teuthology.orchestra.run.smithi179.stderr:election_strategy: 1 2024-03-29T11:19:32.868 INFO:teuthology.orchestra.run.smithi179.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-29T11:19:32.869 INFO:teuthology.orchestra.run.smithi179.stderr:1: v1:172.21.15.179:6789/0 mon.noname-c 2024-03-29T11:19:32.869 INFO:teuthology.orchestra.run.smithi179.stderr:2: v1:172.21.15.84:6790/0 mon.noname-b 2024-03-29T11:19:32.869 INFO:teuthology.orchestra.run.smithi179.stderr: 2024-03-29T11:19:32.878 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:32.892+0000 7fc4a134d640 1 Processor -- start 2024-03-29T11:19:32.878 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:32.892+0000 7fc4a134d640 1 -- start start 2024-03-29T11:19:32.878 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:32.892+0000 7fc4a134d640 10 monclient: init 2024-03-29T11:19:32.878 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:32.892+0000 7fc4a134d640 10 monclient: _reopen_session rank -1 2024-03-29T11:19:32.878 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:32.892+0000 7fc4a134d640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:19:32.878 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:32.892+0000 7fc4a134d640 10 monclient(hunting): picked mon.noname-a con 0x7fc49c09ac40 addr v1:172.21.15.84:6789/0 2024-03-29T11:19:32.878 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:32.892+0000 7fc4a134d640 10 monclient(hunting): picked mon.noname-b con 0x7fc49c09c870 addr v1:172.21.15.84:6790/0 2024-03-29T11:19:32.878 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:32.892+0000 7fc4a134d640 10 monclient(hunting): picked mon.noname-c con 0x7fc49c0b4360 addr v1:172.21.15.179:6789/0 2024-03-29T11:19:32.878 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:32.892+0000 7fc4a134d640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc49c09dec0 con 0x7fc49c09ac40 2024-03-29T11:19:32.878 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:32.892+0000 7fc4a134d640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc49c09da30 con 0x7fc49c0b4360 2024-03-29T11:19:32.879 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:32.892+0000 7fc4a134d640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc49c0b39c0 con 0x7fc49c09c870 2024-03-29T11:19:32.879 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:32.892+0000 7fc4a134d640 10 monclient(hunting): _renew_subs 2024-03-29T11:19:32.879 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:32.892+0000 7fc4a134d640 10 monclient(hunting): authenticate will time out at 811.225082s 2024-03-29T11:19:32.879 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:32.893+0000 7fc49b7fe640 1 --1- >> v1:172.21.15.179:6789/0 conn(0x7fc49c0b4360 0x7fc49c14d9e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.179:6789/0 says I am v1:172.21.15.179:53400/0 (socket says 172.21.15.179:53400) 2024-03-29T11:19:32.879 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:32.893+0000 7fc49b7fe640 1 -- 172.21.15.179:0/2825368046 learned_addr learned my addr 172.21.15.179:0/2825368046 (peer_addr_for_me v1:172.21.15.179:0/0) 2024-03-29T11:19:32.879 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:32.893+0000 7fc499ffb640 1 -- 172.21.15.179:0/2825368046 <== mon.1 v1:172.21.15.179:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2672592911 0 0) 0x7fc49c09da30 con 0x7fc49c0b4360 2024-03-29T11:19:32.879 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:32.893+0000 7fc499ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:19:32.879 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:32.893+0000 7fc499ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:32.879 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:32.893+0000 7fc499ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:32.879 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:32.893+0000 7fc499ffb640 10 monclient(hunting): my global_id is 4105 2024-03-29T11:19:32.879 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:32.893+0000 7fc499ffb640 1 -- 172.21.15.179:0/2825368046 --> v1:172.21.15.179:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc484003520 con 0x7fc49c0b4360 2024-03-29T11:19:32.879 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:32.893+0000 7fc499ffb640 1 -- 172.21.15.179:0/2825368046 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2309983907 0 0) 0x7fc49c0b39c0 con 0x7fc49c09c870 2024-03-29T11:19:32.879 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:32.893+0000 7fc499ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:19:32.879 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:32.893+0000 7fc499ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:32.879 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:32.893+0000 7fc499ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:32.879 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:32.893+0000 7fc499ffb640 10 monclient(hunting): my global_id is 4121 2024-03-29T11:19:32.879 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:32.893+0000 7fc499ffb640 1 -- 172.21.15.179:0/2825368046 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc49c09da30 con 0x7fc49c09c870 2024-03-29T11:19:32.880 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:32.893+0000 7fc499ffb640 1 -- 172.21.15.179:0/2825368046 <== mon.0 v1:172.21.15.84:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1008431734 0 0) 0x7fc490001110 con 0x7fc49c09ac40 2024-03-29T11:19:32.880 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:32.893+0000 7fc499ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:19:32.880 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:32.893+0000 7fc499ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:32.880 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:32.893+0000 7fc499ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:32.880 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:32.893+0000 7fc499ffb640 10 monclient(hunting): my global_id is 4122 2024-03-29T11:19:32.880 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:32.893+0000 7fc499ffb640 1 -- 172.21.15.179:0/2825368046 --> v1:172.21.15.84:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc49c0b39c0 con 0x7fc49c09ac40 2024-03-29T11:19:32.880 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:32.893+0000 7fc499ffb640 1 -- 172.21.15.179:0/2825368046 <== mon.1 v1:172.21.15.179:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3053332655 0 0) 0x7fc484003520 con 0x7fc49c0b4360 2024-03-29T11:19:32.880 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:32.893+0000 7fc499ffb640 1 -- 172.21.15.179:0/2825368046 >> v1:172.21.15.84:6790/0 conn(0x7fc49c09c870 legacy=0x7fc49c13f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:32.880 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:32.893+0000 7fc499ffb640 1 -- 172.21.15.179:0/2825368046 >> v1:172.21.15.84:6789/0 conn(0x7fc49c09ac40 legacy=0x7fc49c0b4a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:32.880 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:32.893+0000 7fc499ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:19:32.880 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:32.893+0000 7fc499ffb640 1 monclient: found mon.noname-c 2024-03-29T11:19:32.880 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:32.893+0000 7fc499ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:19:32.880 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:32.893+0000 7fc499ffb640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.179:6789/0 2024-03-29T11:19:32.880 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:32.893+0000 7fc499ffb640 1 -- 172.21.15.179:0/2825368046 --> v1:172.21.15.179:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc49c0b5160 con 0x7fc49c0b4360 2024-03-29T11:19:32.880 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:32.893+0000 7fc499ffb640 10 monclient: _finish_auth 0 2024-03-29T11:19:32.880 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:32.893+0000 7fc499ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:19:32.881 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:32.893+0000 7fc499ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:19:32.881 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:32.893+0000 7fc499ffb640 1 -- 172.21.15.179:0/2825368046 <== mon.1 v1:172.21.15.179:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4236868694 0 0) 0x7fc494001a10 con 0x7fc49c0b4360 2024-03-29T11:19:32.881 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:32.893+0000 7fc4a134d640 5 monclient: authenticate success, global_id 4105 2024-03-29T11:19:32.881 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:32.893+0000 7fc499ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:32.881 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:32.893+0000 7fc499ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:19:32.881 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:32.893+0000 7fc499ffb640 10 monclient: dump: 2024-03-29T11:19:32.881 INFO:teuthology.orchestra.run.smithi179.stderr:epoch 1 2024-03-29T11:19:32.881 INFO:teuthology.orchestra.run.smithi179.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:19:32.881 INFO:teuthology.orchestra.run.smithi179.stderr:last_changed 2024-03-29T11:18:53.864025+0000 2024-03-29T11:19:32.881 INFO:teuthology.orchestra.run.smithi179.stderr:created 2024-03-29T11:18:53.864025+0000 2024-03-29T11:19:32.881 INFO:teuthology.orchestra.run.smithi179.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:32.881 INFO:teuthology.orchestra.run.smithi179.stderr:election_strategy: 1 2024-03-29T11:19:32.881 INFO:teuthology.orchestra.run.smithi179.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T11:19:32.881 INFO:teuthology.orchestra.run.smithi179.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T11:19:32.881 INFO:teuthology.orchestra.run.smithi179.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T11:19:32.882 INFO:teuthology.orchestra.run.smithi179.stderr: 2024-03-29T11:19:32.882 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:32.893+0000 7fc4a134d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:19:32.882 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:32.893+0000 7fc499ffb640 1 -- 172.21.15.179:0/2825368046 <== mon.1 v1:172.21.15.179:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc494001bf0 con 0x7fc49c0b4360 2024-03-29T11:19:32.882 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:32.893+0000 7fc499ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:19:32.882 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:32.893+0000 7fc499ffb640 1 -- 172.21.15.179:0/2825368046 <== mon.1 v1:172.21.15.179:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4236868694 0 0) 0x7fc494002040 con 0x7fc49c0b4360 2024-03-29T11:19:32.882 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:32.893+0000 7fc499ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:32.882 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:32.893+0000 7fc499ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:19:32.882 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:32.893+0000 7fc499ffb640 10 monclient: dump: 2024-03-29T11:19:32.882 INFO:teuthology.orchestra.run.smithi179.stderr:epoch 1 2024-03-29T11:19:32.882 INFO:teuthology.orchestra.run.smithi179.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:19:32.882 INFO:teuthology.orchestra.run.smithi179.stderr:last_changed 2024-03-29T11:18:53.864025+0000 2024-03-29T11:19:32.882 INFO:teuthology.orchestra.run.smithi179.stderr:created 2024-03-29T11:18:53.864025+0000 2024-03-29T11:19:32.883 INFO:teuthology.orchestra.run.smithi179.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:32.883 INFO:teuthology.orchestra.run.smithi179.stderr:election_strategy: 1 2024-03-29T11:19:32.883 INFO:teuthology.orchestra.run.smithi179.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T11:19:32.883 INFO:teuthology.orchestra.run.smithi179.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T11:19:32.883 INFO:teuthology.orchestra.run.smithi179.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T11:19:32.883 INFO:teuthology.orchestra.run.smithi179.stderr: 2024-03-29T11:19:32.883 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:32.893+0000 7fc4a134d640 10 monclient: get_monmap_and_config success 2024-03-29T11:19:32.883 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:32.893+0000 7fc4a134d640 4 set_mon_vals no callback set 2024-03-29T11:19:32.883 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:32.894+0000 7fc4a134d640 10 monclient: shutdown 2024-03-29T11:19:32.883 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:32.894+0000 7fc4a134d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:19:32.883 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:32.894+0000 7fc4a134d640 1 -- 172.21.15.179:0/2825368046 >> v1:172.21.15.179:6789/0 conn(0x7fc49c0b4360 legacy=0x7fc49c14d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:32.883 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:32.894+0000 7fc4a134d640 1 -- 172.21.15.179:0/2825368046 shutdown_connections 2024-03-29T11:19:32.883 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:32.894+0000 7fc4a134d640 1 -- 172.21.15.179:0/2825368046 >> 172.21.15.179:0/2825368046 conn(0x7fc49c066b00 msgr2=0x7fc49c066f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:19:32.883 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:32.894+0000 7fc4a134d640 1 -- 172.21.15.179:0/2825368046 shutdown_connections 2024-03-29T11:19:32.883 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:32.894+0000 7fc4a134d640 1 -- 172.21.15.179:0/2825368046 wait complete. 2024-03-29T11:19:32.883 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:32.895+0000 7fc4a134d640 10 monclient: build_initial_monmap 2024-03-29T11:19:32.883 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:32.895+0000 7fc4a134d640 10 monclient: monmap: 2024-03-29T11:19:32.884 INFO:teuthology.orchestra.run.smithi179.stderr:epoch 0 2024-03-29T11:19:32.884 INFO:teuthology.orchestra.run.smithi179.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:19:32.884 INFO:teuthology.orchestra.run.smithi179.stderr:last_changed 0.000000 2024-03-29T11:19:32.884 INFO:teuthology.orchestra.run.smithi179.stderr:created 0.000000 2024-03-29T11:19:32.884 INFO:teuthology.orchestra.run.smithi179.stderr:min_mon_release 0 (unknown) 2024-03-29T11:19:32.884 INFO:teuthology.orchestra.run.smithi179.stderr:election_strategy: 1 2024-03-29T11:19:32.884 INFO:teuthology.orchestra.run.smithi179.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-29T11:19:32.884 INFO:teuthology.orchestra.run.smithi179.stderr:1: v1:172.21.15.179:6789/0 mon.noname-b 2024-03-29T11:19:32.884 INFO:teuthology.orchestra.run.smithi179.stderr:2: v1:172.21.15.84:6790/0 mon.noname-c 2024-03-29T11:19:32.884 INFO:teuthology.orchestra.run.smithi179.stderr: 2024-03-29T11:19:32.884 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:32.895+0000 7fc4a134d640 1 librados: starting msgr at 2024-03-29T11:19:32.884 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:32.895+0000 7fc4a134d640 1 librados: starting objecter 2024-03-29T11:19:32.884 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:32.895+0000 7fc4a134d640 1 Processor -- start 2024-03-29T11:19:32.884 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:32.895+0000 7fc4a134d640 1 -- start start 2024-03-29T11:19:32.884 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:32.895+0000 7fc4a134d640 1 librados: setting wanted keys 2024-03-29T11:19:32.884 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:32.895+0000 7fc4a134d640 1 librados: calling monclient init 2024-03-29T11:19:32.885 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:32.895+0000 7fc4a134d640 10 monclient: init 2024-03-29T11:19:32.885 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:32.896+0000 7fc4a134d640 10 monclient: _reopen_session rank -1 2024-03-29T11:19:32.885 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:32.896+0000 7fc4a134d640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:19:32.885 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:32.896+0000 7fc4a134d640 10 monclient(hunting): picked mon.noname-a con 0x7fc49c09ac40 addr v1:172.21.15.84:6789/0 2024-03-29T11:19:32.885 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:32.896+0000 7fc4a134d640 10 monclient(hunting): picked mon.noname-c con 0x7fc49c1462f0 addr v1:172.21.15.84:6790/0 2024-03-29T11:19:32.885 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:32.896+0000 7fc4a134d640 10 monclient(hunting): picked mon.noname-b con 0x7fc49c1401e0 addr v1:172.21.15.179:6789/0 2024-03-29T11:19:32.885 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:32.896+0000 7fc4a134d640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc49c142050 con 0x7fc49c09ac40 2024-03-29T11:19:32.885 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:32.896+0000 7fc4a134d640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc49c0b1200 con 0x7fc49c1401e0 2024-03-29T11:19:32.885 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:32.896+0000 7fc4a134d640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc49c0b1450 con 0x7fc49c1462f0 2024-03-29T11:19:32.885 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:32.896+0000 7fc4a134d640 10 monclient(hunting): _renew_subs 2024-03-29T11:19:32.885 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:32.896+0000 7fc4a134d640 10 monclient(hunting): authenticate will time out at 811.228321s 2024-03-29T11:19:32.885 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:32.896+0000 7fc49b7fe640 1 --1- >> v1:172.21.15.179:6789/0 conn(0x7fc49c1401e0 0x7fc49c139d20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.179:6789/0 says I am v1:172.21.15.179:53412/0 (socket says 172.21.15.179:53412) 2024-03-29T11:19:32.885 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:32.896+0000 7fc49b7fe640 1 -- 172.21.15.179:0/3115065678 learned_addr learned my addr 172.21.15.179:0/3115065678 (peer_addr_for_me v1:172.21.15.179:0/0) 2024-03-29T11:19:32.885 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:32.896+0000 7fc4837fe640 10 client.?.objecter ms_handle_connect 0x7fc49c1401e0 2024-03-29T11:19:32.885 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:32.896+0000 7fc4837fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:19:32.885 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:32.896+0000 7fc4837fe640 10 client.?.objecter ms_handle_connect 0x7fc49c09ac40 2024-03-29T11:19:32.886 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:32.896+0000 7fc4837fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:19:32.886 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:32.896+0000 7fc4837fe640 10 client.?.objecter ms_handle_connect 0x7fc49c1462f0 2024-03-29T11:19:32.886 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:32.896+0000 7fc4837fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:19:32.886 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:32.896+0000 7fc4837fe640 1 -- 172.21.15.179:0/3115065678 <== mon.1 v1:172.21.15.179:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2016507128 0 0) 0x7fc49c0b1200 con 0x7fc49c1401e0 2024-03-29T11:19:32.886 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:32.896+0000 7fc4837fe640 10 monclient(hunting): _negotiate 2024-03-29T11:19:32.886 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:32.896+0000 7fc4837fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:32.886 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:32.896+0000 7fc4837fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:32.886 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:32.896+0000 7fc4837fe640 10 monclient(hunting): my global_id is 4108 2024-03-29T11:19:32.886 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:32.896+0000 7fc4837fe640 1 -- 172.21.15.179:0/3115065678 --> v1:172.21.15.179:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc488004eb0 con 0x7fc49c1401e0 2024-03-29T11:19:32.886 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:32.896+0000 7fc4837fe640 1 -- 172.21.15.179:0/3115065678 <== mon.0 v1:172.21.15.84:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1906488466 0 0) 0x7fc49c142050 con 0x7fc49c09ac40 2024-03-29T11:19:32.886 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:32.896+0000 7fc4837fe640 10 monclient(hunting): _negotiate 2024-03-29T11:19:32.886 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:32.896+0000 7fc4837fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:32.886 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:32.896+0000 7fc4837fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:32.886 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:32.896+0000 7fc4837fe640 10 monclient(hunting): my global_id is 4125 2024-03-29T11:19:32.887 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:32.896+0000 7fc4837fe640 1 -- 172.21.15.179:0/3115065678 --> v1:172.21.15.84:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc49c0b1200 con 0x7fc49c09ac40 2024-03-29T11:19:32.887 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:32.896+0000 7fc4837fe640 1 -- 172.21.15.179:0/3115065678 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3184312515 0 0) 0x7fc49c0b1450 con 0x7fc49c1462f0 2024-03-29T11:19:32.887 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:32.896+0000 7fc4837fe640 10 monclient(hunting): _negotiate 2024-03-29T11:19:32.887 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:32.896+0000 7fc4837fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:32.887 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:32.896+0000 7fc4837fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:32.887 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:32.896+0000 7fc4837fe640 10 monclient(hunting): my global_id is 4124 2024-03-29T11:19:32.887 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:32.896+0000 7fc4837fe640 1 -- 172.21.15.179:0/3115065678 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc49c142050 con 0x7fc49c1462f0 2024-03-29T11:19:32.887 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:32.896+0000 7fc4837fe640 1 -- 172.21.15.179:0/3115065678 <== mon.0 v1:172.21.15.84:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 99957095 0 0) 0x7fc49c0b1200 con 0x7fc49c09ac40 2024-03-29T11:19:32.887 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:32.896+0000 7fc4837fe640 1 -- 172.21.15.179:0/3115065678 --> v1:172.21.15.84:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc49c0b1450 con 0x7fc49c09ac40 2024-03-29T11:19:32.887 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:32.896+0000 7fc4837fe640 1 -- 172.21.15.179:0/3115065678 <== mon.0 v1:172.21.15.84:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4236868694 0 0) 0x7fc490001a00 con 0x7fc49c09ac40 2024-03-29T11:19:32.887 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:32.896+0000 7fc4837fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:32.887 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:32.896+0000 7fc4837fe640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:19:32.887 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:32.896+0000 7fc4837fe640 10 monclient(hunting): dump: 2024-03-29T11:19:32.887 INFO:teuthology.orchestra.run.smithi179.stderr:epoch 1 2024-03-29T11:19:32.887 INFO:teuthology.orchestra.run.smithi179.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:19:32.888 INFO:teuthology.orchestra.run.smithi179.stderr:last_changed 2024-03-29T11:18:53.864025+0000 2024-03-29T11:19:32.888 INFO:teuthology.orchestra.run.smithi179.stderr:created 2024-03-29T11:18:53.864025+0000 2024-03-29T11:19:32.888 INFO:teuthology.orchestra.run.smithi179.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:32.888 INFO:teuthology.orchestra.run.smithi179.stderr:election_strategy: 1 2024-03-29T11:19:32.888 INFO:teuthology.orchestra.run.smithi179.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T11:19:32.888 INFO:teuthology.orchestra.run.smithi179.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T11:19:32.888 INFO:teuthology.orchestra.run.smithi179.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T11:19:32.888 INFO:teuthology.orchestra.run.smithi179.stderr: 2024-03-29T11:19:32.888 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:32.896+0000 7fc4837fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:19:32.888 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:32.896+0000 7fc4837fe640 1 -- 172.21.15.179:0/3115065678 <== mon.1 v1:172.21.15.179:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3056836794 0 0) 0x7fc488004eb0 con 0x7fc49c1401e0 2024-03-29T11:19:32.888 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:32.896+0000 7fc4837fe640 1 -- 172.21.15.179:0/3115065678 --> v1:172.21.15.179:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc49c0b1200 con 0x7fc49c1401e0 2024-03-29T11:19:32.888 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:32.896+0000 7fc4837fe640 1 -- 172.21.15.179:0/3115065678 <== mon.1 v1:172.21.15.179:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4236868694 0 0) 0x7fc494001f00 con 0x7fc49c1401e0 2024-03-29T11:19:32.888 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:32.896+0000 7fc4837fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:32.888 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:32.896+0000 7fc4837fe640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:19:32.888 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:32.896+0000 7fc4837fe640 10 monclient(hunting): dump: 2024-03-29T11:19:32.889 INFO:teuthology.orchestra.run.smithi179.stderr:epoch 1 2024-03-29T11:19:32.889 INFO:teuthology.orchestra.run.smithi179.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:19:32.889 INFO:teuthology.orchestra.run.smithi179.stderr:last_changed 2024-03-29T11:18:53.864025+0000 2024-03-29T11:19:32.889 INFO:teuthology.orchestra.run.smithi179.stderr:created 2024-03-29T11:18:53.864025+0000 2024-03-29T11:19:32.889 INFO:teuthology.orchestra.run.smithi179.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:32.889 INFO:teuthology.orchestra.run.smithi179.stderr:election_strategy: 1 2024-03-29T11:19:32.889 INFO:teuthology.orchestra.run.smithi179.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T11:19:32.889 INFO:teuthology.orchestra.run.smithi179.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T11:19:32.889 INFO:teuthology.orchestra.run.smithi179.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T11:19:32.889 INFO:teuthology.orchestra.run.smithi179.stderr: 2024-03-29T11:19:32.889 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:32.896+0000 7fc4837fe640 1 -- 172.21.15.179:0/3115065678 <== mon.2 v1:172.21.15.84:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1413549444 0 0) 0x7fc49c142050 con 0x7fc49c1462f0 2024-03-29T11:19:32.889 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:32.896+0000 7fc4837fe640 1 -- 172.21.15.179:0/3115065678 --> v1:172.21.15.84:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc488004eb0 con 0x7fc49c1462f0 2024-03-29T11:19:32.889 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:32.896+0000 7fc4837fe640 1 -- 172.21.15.179:0/3115065678 <== mon.2 v1:172.21.15.84:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4236868694 0 0) 0x7fc48c001750 con 0x7fc49c1462f0 2024-03-29T11:19:32.889 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:32.896+0000 7fc4837fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:32.889 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:32.896+0000 7fc4837fe640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:19:32.890 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:32.896+0000 7fc4837fe640 10 monclient(hunting): dump: 2024-03-29T11:19:32.890 INFO:teuthology.orchestra.run.smithi179.stderr:epoch 1 2024-03-29T11:19:32.890 INFO:teuthology.orchestra.run.smithi179.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:19:32.890 INFO:teuthology.orchestra.run.smithi179.stderr:last_changed 2024-03-29T11:18:53.864025+0000 2024-03-29T11:19:32.890 INFO:teuthology.orchestra.run.smithi179.stderr:created 2024-03-29T11:18:53.864025+0000 2024-03-29T11:19:32.890 INFO:teuthology.orchestra.run.smithi179.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:32.890 INFO:teuthology.orchestra.run.smithi179.stderr:election_strategy: 1 2024-03-29T11:19:32.890 INFO:teuthology.orchestra.run.smithi179.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T11:19:32.890 INFO:teuthology.orchestra.run.smithi179.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T11:19:32.890 INFO:teuthology.orchestra.run.smithi179.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T11:19:32.890 INFO:teuthology.orchestra.run.smithi179.stderr: 2024-03-29T11:19:32.890 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:32.896+0000 7fc4837fe640 1 -- 172.21.15.179:0/3115065678 <== mon.1 v1:172.21.15.179:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 656279162 0 0) 0x7fc49c0b1200 con 0x7fc49c1401e0 2024-03-29T11:19:32.890 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:32.896+0000 7fc4837fe640 1 -- 172.21.15.179:0/3115065678 >> v1:172.21.15.84:6790/0 conn(0x7fc49c1462f0 legacy=0x7fc49c140780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:32.890 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:32.896+0000 7fc4837fe640 1 -- 172.21.15.179:0/3115065678 >> v1:172.21.15.84:6789/0 conn(0x7fc49c09ac40 legacy=0x7fc49c142a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:32.891 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:32.896+0000 7fc4837fe640 10 monclient: _finish_hunting 0 2024-03-29T11:19:32.891 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:32.896+0000 7fc4837fe640 1 monclient: found mon.b 2024-03-29T11:19:32.891 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:32.896+0000 7fc4837fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:19:32.891 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:32.896+0000 7fc4837fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-29T11:19:32.891 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:32.896+0000 7fc4837fe640 1 -- 172.21.15.179:0/3115065678 --> v1:172.21.15.179:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc49c143120 con 0x7fc49c1401e0 2024-03-29T11:19:32.891 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:32.897+0000 7fc4837fe640 10 monclient: _finish_auth 0 2024-03-29T11:19:32.891 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:32.897+0000 7fc4837fe640 10 monclient: _check_auth_tickets 2024-03-29T11:19:32.891 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:32.897+0000 7fc4837fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:19:32.891 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:32.897+0000 7fc4a134d640 5 monclient: authenticate success, global_id 4108 2024-03-29T11:19:32.891 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:32.897+0000 7fc4a134d640 10 monclient: _renew_subs 2024-03-29T11:19:32.891 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:32.897+0000 7fc4a134d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-29T11:19:32.891 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:32.897+0000 7fc4a134d640 1 -- 172.21.15.179:0/3115065678 --> v1:172.21.15.179:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc49c140e90 con 0x7fc49c1401e0 2024-03-29T11:19:32.891 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:32.897+0000 7fc4a134d640 10 client.4108.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:19:32.891 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:32.897+0000 7fc4a134d640 10 monclient: _renew_subs 2024-03-29T11:19:32.891 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:32.897+0000 7fc4a134d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-29T11:19:32.891 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:32.897+0000 7fc4a134d640 1 -- 172.21.15.179:0/3115065678 --> v1:172.21.15.179:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc49c13a610 con 0x7fc49c1401e0 2024-03-29T11:19:32.892 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:32.897+0000 7fc4a134d640 1 librados: init done 2024-03-29T11:19:32.892 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:32.897+0000 7fc4837fe640 1 -- 172.21.15.179:0/3115065678 <== mon.1 v1:172.21.15.179:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc494004e10 con 0x7fc49c1401e0 2024-03-29T11:19:32.892 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:32.897+0000 7fc4837fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:19:32.892 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:32.897+0000 7fc4837fe640 1 -- 172.21.15.179:0/3115065678 <== mon.1 v1:172.21.15.179:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4236868694 0 0) 0x7fc494002100 con 0x7fc49c1401e0 2024-03-29T11:19:32.892 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:32.897+0000 7fc4837fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:32.892 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:32.897+0000 7fc4837fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:19:32.892 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:32.897+0000 7fc4997fa640 4 set_mon_vals no callback set 2024-03-29T11:19:32.892 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:32.897+0000 7fc4837fe640 10 monclient: dump: 2024-03-29T11:19:32.892 INFO:teuthology.orchestra.run.smithi179.stderr:epoch 1 2024-03-29T11:19:32.892 INFO:teuthology.orchestra.run.smithi179.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:19:32.892 INFO:teuthology.orchestra.run.smithi179.stderr:last_changed 2024-03-29T11:18:53.864025+0000 2024-03-29T11:19:32.892 INFO:teuthology.orchestra.run.smithi179.stderr:created 2024-03-29T11:18:53.864025+0000 2024-03-29T11:19:32.892 INFO:teuthology.orchestra.run.smithi179.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:32.892 INFO:teuthology.orchestra.run.smithi179.stderr:election_strategy: 1 2024-03-29T11:19:32.892 INFO:teuthology.orchestra.run.smithi179.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T11:19:32.893 INFO:teuthology.orchestra.run.smithi179.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T11:19:32.893 INFO:teuthology.orchestra.run.smithi179.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T11:19:32.893 INFO:teuthology.orchestra.run.smithi179.stderr: 2024-03-29T11:19:32.893 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:32.897+0000 7fc4837fe640 1 -- 172.21.15.179:0/3115065678 <== mon.1 v1:172.21.15.179:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7fc4940054b0 con 0x7fc49c1401e0 2024-03-29T11:19:32.893 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:32.897+0000 7fc4837fe640 1 -- 172.21.15.179:0/3115065678 <== mon.1 v1:172.21.15.179:6789/0 8 ==== osd_map(3..3 src has 1..3) v4 ==== 840+0+0 (unknown 23745654 0 0) 0x7fc494006a40 con 0x7fc49c1401e0 2024-03-29T11:19:32.893 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:32.897+0000 7fc4837fe640 10 client.4108.objecter ms_dispatch 0x7fc49c000b90 osd_map(3..3 src has 1..3) v4 2024-03-29T11:19:32.893 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:32.897+0000 7fc4837fe640 3 client.4108.objecter handle_osd_map got epochs [3,3] > 0 2024-03-29T11:19:32.893 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:32.897+0000 7fc4837fe640 3 client.4108.objecter handle_osd_map decoding full epoch 3 2024-03-29T11:19:32.893 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:32.897+0000 7fc4837fe640 20 client.4108.objecter dump_active .. 0 homeless 2024-03-29T11:19:32.893 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:32.897+0000 7fc4a134d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:32.893 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:32.897+0000 7fc4a134d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:32.893 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:32.897+0000 7fc4a134d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-29T11:19:32.893 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:32.897+0000 7fc4a134d640 1 -- 172.21.15.179:0/3115065678 --> v1:172.21.15.179:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc47c005080 con 0x7fc49c1401e0 2024-03-29T11:19:32.893 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:32.900+0000 7fc4837fe640 1 -- 172.21.15.179:0/3115065678 <== mon.1 v1:172.21.15.179:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7fc494005940 con 0x7fc49c1401e0 2024-03-29T11:19:32.893 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:32.900+0000 7fc4837fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:32.893 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:32.900+0000 7fc4837fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:19:32.948 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:32.971+0000 7fc4a134d640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "cd580e99-109f-4bf1-88c0-a5987bc8b9ed", "id": 1}] 2024-03-29T11:19:32.948 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:32.971+0000 7fc4a134d640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "cd580e99-109f-4bf1-88c0-a5987bc8b9ed", "id": 1}] 2024-03-29T11:19:32.948 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:32.971+0000 7fc4a134d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-29T11:19:32.948 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:32.971+0000 7fc4a134d640 1 -- 172.21.15.179:0/3115065678 --> v1:172.21.15.179:6789/0 -- mon_command({"prefix": "osd new", "uuid": "cd580e99-109f-4bf1-88c0-a5987bc8b9ed", "id": 1} v 0) v1 -- 0x7fc47c0058b0 con 0x7fc49c1401e0 2024-03-29T11:19:32.956 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:32.979+0000 7fc4837fe640 1 -- 172.21.15.179:0/3115065678 <== mon.1 v1:172.21.15.179:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "cd580e99-109f-4bf1-88c0-a5987bc8b9ed", "id": 1}]=0 v4) v1 ==== 112+0+2 (unknown 2340394762 0 940483276) 0x7fc494005c90 con 0x7fc49c1401e0 2024-03-29T11:19:32.956 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:32.979+0000 7fc4837fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "cd580e99-109f-4bf1-88c0-a5987bc8b9ed", "id": 1}] 2024-03-29T11:19:32.956 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:32.979+0000 7fc4837fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:19:32.956 INFO:teuthology.orchestra.run.smithi179.stdout:1 2024-03-29T11:19:32.957 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:32.980+0000 7fc4a134d640 10 librados: watch_flush enter 2024-03-29T11:19:32.957 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:32.980+0000 7fc4a134d640 10 librados: watch_flush exit 2024-03-29T11:19:32.957 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:32.980+0000 7fc4a134d640 20 client.4108.objecter shutdown clearing up homeless session... 2024-03-29T11:19:32.957 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:32.980+0000 7fc4a134d640 10 client.4108.objecter successfully canceled tick 2024-03-29T11:19:32.957 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:32.980+0000 7fc4a134d640 10 monclient: shutdown 2024-03-29T11:19:32.958 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:32.980+0000 7fc4a134d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:19:32.958 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:32.980+0000 7fc4a134d640 1 -- 172.21.15.179:0/3115065678 >> v1:172.21.15.179:6789/0 conn(0x7fc49c1401e0 legacy=0x7fc49c139d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:32.958 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:32.980+0000 7fc4a134d640 1 -- 172.21.15.179:0/3115065678 shutdown_connections 2024-03-29T11:19:32.958 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:32.980+0000 7fc4a134d640 1 -- 172.21.15.179:0/3115065678 >> 172.21.15.179:0/3115065678 conn(0x7fc49c066b00 msgr2=0x7fc49c0b4a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:19:32.958 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:32.980+0000 7fc4a134d640 1 -- 172.21.15.179:0/3115065678 shutdown_connections 2024-03-29T11:19:32.958 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:32.980+0000 7fc4a134d640 1 -- 172.21.15.179:0/3115065678 wait complete. 2024-03-29T11:19:32.958 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:32.980+0000 7fc4a134d640 1 librados: shutdown 2024-03-29T11:19:32.967 DEBUG:teuthology.orchestra.run.smithi179:> sudo ceph --cluster ceph osd new 11e60a5a-051d-453f-a190-66387cb35592 2 2024-03-29T11:19:33.069 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.092+0000 7efc8d3d6640 10 monclient: get_monmap_and_config 2024-03-29T11:19:33.069 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.092+0000 7efc8d3d6640 10 monclient: build_initial_monmap 2024-03-29T11:19:33.069 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.092+0000 7efc8d3d6640 10 monclient: monmap: 2024-03-29T11:19:33.069 INFO:teuthology.orchestra.run.smithi179.stderr:epoch 0 2024-03-29T11:19:33.069 INFO:teuthology.orchestra.run.smithi179.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:19:33.069 INFO:teuthology.orchestra.run.smithi179.stderr:last_changed 2024-03-29T11:19:33.093019+0000 2024-03-29T11:19:33.069 INFO:teuthology.orchestra.run.smithi179.stderr:created 2024-03-29T11:19:33.093019+0000 2024-03-29T11:19:33.069 INFO:teuthology.orchestra.run.smithi179.stderr:min_mon_release 0 (unknown) 2024-03-29T11:19:33.069 INFO:teuthology.orchestra.run.smithi179.stderr:election_strategy: 1 2024-03-29T11:19:33.069 INFO:teuthology.orchestra.run.smithi179.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-29T11:19:33.069 INFO:teuthology.orchestra.run.smithi179.stderr:1: v1:172.21.15.179:6789/0 mon.noname-c 2024-03-29T11:19:33.069 INFO:teuthology.orchestra.run.smithi179.stderr:2: v1:172.21.15.84:6790/0 mon.noname-b 2024-03-29T11:19:33.069 INFO:teuthology.orchestra.run.smithi179.stderr: 2024-03-29T11:19:33.070 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.092+0000 7efc8d3d6640 1 Processor -- start 2024-03-29T11:19:33.070 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.092+0000 7efc8d3d6640 1 -- start start 2024-03-29T11:19:33.070 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.092+0000 7efc8d3d6640 10 monclient: init 2024-03-29T11:19:33.078 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.093+0000 7efc8d3d6640 10 monclient: _reopen_session rank -1 2024-03-29T11:19:33.078 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.093+0000 7efc8d3d6640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:19:33.078 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.093+0000 7efc8d3d6640 10 monclient(hunting): picked mon.noname-c con 0x7efc8809ac40 addr v1:172.21.15.179:6789/0 2024-03-29T11:19:33.078 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.093+0000 7efc8d3d6640 10 monclient(hunting): picked mon.noname-a con 0x7efc8809c870 addr v1:172.21.15.84:6789/0 2024-03-29T11:19:33.078 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.093+0000 7efc8d3d6640 10 monclient(hunting): picked mon.noname-b con 0x7efc880b4360 addr v1:172.21.15.84:6790/0 2024-03-29T11:19:33.078 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.093+0000 7efc8d3d6640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efc8809dec0 con 0x7efc8809c870 2024-03-29T11:19:33.079 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.093+0000 7efc8d3d6640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efc8809da30 con 0x7efc8809ac40 2024-03-29T11:19:33.079 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.093+0000 7efc8d3d6640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efc880b39c0 con 0x7efc880b4360 2024-03-29T11:19:33.079 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.093+0000 7efc8d3d6640 10 monclient(hunting): _renew_subs 2024-03-29T11:19:33.079 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.093+0000 7efc8d3d6640 10 monclient(hunting): authenticate will time out at 811.425413s 2024-03-29T11:19:33.079 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.093+0000 7efc86ffd640 1 --1- >> v1:172.21.15.179:6789/0 conn(0x7efc8809ac40 0x7efc880b4a50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.179:6789/0 says I am v1:172.21.15.179:53428/0 (socket says 172.21.15.179:53428) 2024-03-29T11:19:33.079 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.093+0000 7efc86ffd640 1 -- 172.21.15.179:0/3471544010 learned_addr learned my addr 172.21.15.179:0/3471544010 (peer_addr_for_me v1:172.21.15.179:0/0) 2024-03-29T11:19:33.079 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.093+0000 7efc85ffb640 1 -- 172.21.15.179:0/3471544010 <== mon.1 v1:172.21.15.179:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1330886251 0 0) 0x7efc8809da30 con 0x7efc8809ac40 2024-03-29T11:19:33.079 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.093+0000 7efc85ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:19:33.079 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.093+0000 7efc85ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:33.079 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.093+0000 7efc85ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:33.079 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.093+0000 7efc85ffb640 10 monclient(hunting): my global_id is 4111 2024-03-29T11:19:33.079 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.093+0000 7efc85ffb640 1 -- 172.21.15.179:0/3471544010 --> v1:172.21.15.179:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efc70003520 con 0x7efc8809ac40 2024-03-29T11:19:33.079 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.093+0000 7efc85ffb640 1 -- 172.21.15.179:0/3471544010 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1954032275 0 0) 0x7efc880b39c0 con 0x7efc880b4360 2024-03-29T11:19:33.079 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.093+0000 7efc85ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:19:33.079 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.093+0000 7efc85ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:33.079 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.093+0000 7efc85ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:33.080 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.093+0000 7efc85ffb640 10 monclient(hunting): my global_id is 4127 2024-03-29T11:19:33.080 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.093+0000 7efc85ffb640 1 -- 172.21.15.179:0/3471544010 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efc8809da30 con 0x7efc880b4360 2024-03-29T11:19:33.080 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.093+0000 7efc85ffb640 1 -- 172.21.15.179:0/3471544010 <== mon.0 v1:172.21.15.84:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3526501628 0 0) 0x7efc78001110 con 0x7efc8809c870 2024-03-29T11:19:33.080 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.093+0000 7efc85ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:19:33.080 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.093+0000 7efc85ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:33.080 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.093+0000 7efc85ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:33.080 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.093+0000 7efc85ffb640 10 monclient(hunting): my global_id is 4128 2024-03-29T11:19:33.080 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.093+0000 7efc85ffb640 1 -- 172.21.15.179:0/3471544010 --> v1:172.21.15.84:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efc880b39c0 con 0x7efc8809c870 2024-03-29T11:19:33.080 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.093+0000 7efc85ffb640 1 -- 172.21.15.179:0/3471544010 <== mon.1 v1:172.21.15.179:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1836352263 0 0) 0x7efc70003520 con 0x7efc8809ac40 2024-03-29T11:19:33.080 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.093+0000 7efc85ffb640 1 -- 172.21.15.179:0/3471544010 >> v1:172.21.15.84:6790/0 conn(0x7efc880b4360 legacy=0x7efc8814d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:33.080 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.093+0000 7efc85ffb640 1 -- 172.21.15.179:0/3471544010 >> v1:172.21.15.84:6789/0 conn(0x7efc8809c870 legacy=0x7efc8813f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:33.080 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.093+0000 7efc85ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:19:33.080 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.093+0000 7efc85ffb640 1 monclient: found mon.noname-c 2024-03-29T11:19:33.080 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.093+0000 7efc85ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:19:33.080 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.093+0000 7efc85ffb640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.179:6789/0 2024-03-29T11:19:33.081 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.093+0000 7efc85ffb640 1 -- 172.21.15.179:0/3471544010 --> v1:172.21.15.179:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc880b5160 con 0x7efc8809ac40 2024-03-29T11:19:33.081 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.093+0000 7efc85ffb640 10 monclient: _finish_auth 0 2024-03-29T11:19:33.081 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.093+0000 7efc85ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:19:33.081 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.093+0000 7efc85ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:19:33.081 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.093+0000 7efc8d3d6640 5 monclient: authenticate success, global_id 4111 2024-03-29T11:19:33.081 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.093+0000 7efc85ffb640 1 -- 172.21.15.179:0/3471544010 <== mon.1 v1:172.21.15.179:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4236868694 0 0) 0x7efc7c001690 con 0x7efc8809ac40 2024-03-29T11:19:33.081 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.093+0000 7efc8d3d6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:19:33.081 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.093+0000 7efc85ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:33.081 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.093+0000 7efc85ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:19:33.081 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.093+0000 7efc85ffb640 10 monclient: dump: 2024-03-29T11:19:33.081 INFO:teuthology.orchestra.run.smithi179.stderr:epoch 1 2024-03-29T11:19:33.081 INFO:teuthology.orchestra.run.smithi179.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:19:33.081 INFO:teuthology.orchestra.run.smithi179.stderr:last_changed 2024-03-29T11:18:53.864025+0000 2024-03-29T11:19:33.081 INFO:teuthology.orchestra.run.smithi179.stderr:created 2024-03-29T11:18:53.864025+0000 2024-03-29T11:19:33.081 INFO:teuthology.orchestra.run.smithi179.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:33.081 INFO:teuthology.orchestra.run.smithi179.stderr:election_strategy: 1 2024-03-29T11:19:33.081 INFO:teuthology.orchestra.run.smithi179.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T11:19:33.082 INFO:teuthology.orchestra.run.smithi179.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T11:19:33.082 INFO:teuthology.orchestra.run.smithi179.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T11:19:33.082 INFO:teuthology.orchestra.run.smithi179.stderr: 2024-03-29T11:19:33.082 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.093+0000 7efc8d3d6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:19:33.082 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.094+0000 7efc85ffb640 1 -- 172.21.15.179:0/3471544010 <== mon.1 v1:172.21.15.179:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7efc7c001870 con 0x7efc8809ac40 2024-03-29T11:19:33.082 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.094+0000 7efc85ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:19:33.082 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.094+0000 7efc85ffb640 1 -- 172.21.15.179:0/3471544010 <== mon.1 v1:172.21.15.179:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4236868694 0 0) 0x7efc7c001c80 con 0x7efc8809ac40 2024-03-29T11:19:33.082 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.094+0000 7efc8d3d6640 10 monclient: get_monmap_and_config success 2024-03-29T11:19:33.082 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.094+0000 7efc8d3d6640 4 set_mon_vals no callback set 2024-03-29T11:19:33.082 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.094+0000 7efc85ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:33.082 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.094+0000 7efc85ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:19:33.082 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.094+0000 7efc85ffb640 10 monclient: dump: 2024-03-29T11:19:33.082 INFO:teuthology.orchestra.run.smithi179.stderr:epoch 1 2024-03-29T11:19:33.082 INFO:teuthology.orchestra.run.smithi179.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:19:33.083 INFO:teuthology.orchestra.run.smithi179.stderr:last_changed 2024-03-29T11:18:53.864025+0000 2024-03-29T11:19:33.083 INFO:teuthology.orchestra.run.smithi179.stderr:created 2024-03-29T11:18:53.864025+0000 2024-03-29T11:19:33.083 INFO:teuthology.orchestra.run.smithi179.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:33.083 INFO:teuthology.orchestra.run.smithi179.stderr:election_strategy: 1 2024-03-29T11:19:33.083 INFO:teuthology.orchestra.run.smithi179.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T11:19:33.083 INFO:teuthology.orchestra.run.smithi179.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T11:19:33.083 INFO:teuthology.orchestra.run.smithi179.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T11:19:33.083 INFO:teuthology.orchestra.run.smithi179.stderr: 2024-03-29T11:19:33.083 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.094+0000 7efc8d3d6640 10 monclient: shutdown 2024-03-29T11:19:33.083 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.094+0000 7efc8d3d6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:19:33.083 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.094+0000 7efc8d3d6640 1 -- 172.21.15.179:0/3471544010 >> v1:172.21.15.179:6789/0 conn(0x7efc8809ac40 legacy=0x7efc880b4a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:33.083 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.094+0000 7efc8d3d6640 1 -- 172.21.15.179:0/3471544010 shutdown_connections 2024-03-29T11:19:33.083 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.094+0000 7efc8d3d6640 1 -- 172.21.15.179:0/3471544010 >> 172.21.15.179:0/3471544010 conn(0x7efc88066b00 msgr2=0x7efc88066f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:19:33.084 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.094+0000 7efc8d3d6640 1 -- 172.21.15.179:0/3471544010 shutdown_connections 2024-03-29T11:19:33.084 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.094+0000 7efc8d3d6640 1 -- 172.21.15.179:0/3471544010 wait complete. 2024-03-29T11:19:33.084 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.095+0000 7efc8d3d6640 10 monclient: build_initial_monmap 2024-03-29T11:19:33.084 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.095+0000 7efc8d3d6640 10 monclient: monmap: 2024-03-29T11:19:33.084 INFO:teuthology.orchestra.run.smithi179.stderr:epoch 0 2024-03-29T11:19:33.084 INFO:teuthology.orchestra.run.smithi179.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:19:33.084 INFO:teuthology.orchestra.run.smithi179.stderr:last_changed 0.000000 2024-03-29T11:19:33.084 INFO:teuthology.orchestra.run.smithi179.stderr:created 0.000000 2024-03-29T11:19:33.084 INFO:teuthology.orchestra.run.smithi179.stderr:min_mon_release 0 (unknown) 2024-03-29T11:19:33.084 INFO:teuthology.orchestra.run.smithi179.stderr:election_strategy: 1 2024-03-29T11:19:33.084 INFO:teuthology.orchestra.run.smithi179.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-29T11:19:33.084 INFO:teuthology.orchestra.run.smithi179.stderr:1: v1:172.21.15.179:6789/0 mon.noname-b 2024-03-29T11:19:33.084 INFO:teuthology.orchestra.run.smithi179.stderr:2: v1:172.21.15.84:6790/0 mon.noname-c 2024-03-29T11:19:33.084 INFO:teuthology.orchestra.run.smithi179.stderr: 2024-03-29T11:19:33.084 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.095+0000 7efc8d3d6640 1 librados: starting msgr at 2024-03-29T11:19:33.084 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.095+0000 7efc8d3d6640 1 librados: starting objecter 2024-03-29T11:19:33.085 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.095+0000 7efc8d3d6640 1 Processor -- start 2024-03-29T11:19:33.085 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.095+0000 7efc8d3d6640 1 -- start start 2024-03-29T11:19:33.085 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.095+0000 7efc8d3d6640 1 librados: setting wanted keys 2024-03-29T11:19:33.085 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.095+0000 7efc8d3d6640 1 librados: calling monclient init 2024-03-29T11:19:33.085 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.095+0000 7efc8d3d6640 10 monclient: init 2024-03-29T11:19:33.085 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.096+0000 7efc8d3d6640 10 monclient: _reopen_session rank -1 2024-03-29T11:19:33.085 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.096+0000 7efc8d3d6640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:19:33.085 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.096+0000 7efc8d3d6640 10 monclient(hunting): picked mon.noname-a con 0x7efc8809ac40 addr v1:172.21.15.84:6789/0 2024-03-29T11:19:33.085 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.096+0000 7efc8d3d6640 10 monclient(hunting): picked mon.noname-c con 0x7efc880acd70 addr v1:172.21.15.84:6790/0 2024-03-29T11:19:33.085 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.096+0000 7efc8d3d6640 10 monclient(hunting): picked mon.noname-b con 0x7efc88139820 addr v1:172.21.15.179:6789/0 2024-03-29T11:19:33.085 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.096+0000 7efc8d3d6640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efc880b1a60 con 0x7efc8809ac40 2024-03-29T11:19:33.085 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.096+0000 7efc8d3d6640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efc881d73f0 con 0x7efc88139820 2024-03-29T11:19:33.085 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.096+0000 7efc8d3d6640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efc881d8020 con 0x7efc880acd70 2024-03-29T11:19:33.085 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.096+0000 7efc8d3d6640 10 monclient(hunting): _renew_subs 2024-03-29T11:19:33.085 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.096+0000 7efc8d3d6640 10 monclient(hunting): authenticate will time out at 811.428537s 2024-03-29T11:19:33.086 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.096+0000 7efc877fe640 1 --1- >> v1:172.21.15.179:6789/0 conn(0x7efc88139820 0x7efc880b00a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.179:6789/0 says I am v1:172.21.15.179:53436/0 (socket says 172.21.15.179:53436) 2024-03-29T11:19:33.086 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.096+0000 7efc877fe640 1 -- 172.21.15.179:0/4216634740 learned_addr learned my addr 172.21.15.179:0/4216634740 (peer_addr_for_me v1:172.21.15.179:0/0) 2024-03-29T11:19:33.086 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.096+0000 7efc6f7fe640 10 client.?.objecter ms_handle_connect 0x7efc88139820 2024-03-29T11:19:33.086 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.096+0000 7efc6f7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:19:33.086 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.096+0000 7efc6f7fe640 10 client.?.objecter ms_handle_connect 0x7efc8809ac40 2024-03-29T11:19:33.086 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.096+0000 7efc6f7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:19:33.086 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.096+0000 7efc6f7fe640 10 client.?.objecter ms_handle_connect 0x7efc880acd70 2024-03-29T11:19:33.086 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.096+0000 7efc6f7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:19:33.086 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.096+0000 7efc6f7fe640 1 -- 172.21.15.179:0/4216634740 <== mon.1 v1:172.21.15.179:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4203081511 0 0) 0x7efc881d73f0 con 0x7efc88139820 2024-03-29T11:19:33.086 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.096+0000 7efc6f7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:19:33.086 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.096+0000 7efc6f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:33.086 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.096+0000 7efc6f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:33.087 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.096+0000 7efc6f7fe640 10 monclient(hunting): my global_id is 4114 2024-03-29T11:19:33.087 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.096+0000 7efc6f7fe640 1 -- 172.21.15.179:0/4216634740 --> v1:172.21.15.179:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efc74004eb0 con 0x7efc88139820 2024-03-29T11:19:33.087 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.096+0000 7efc6f7fe640 1 -- 172.21.15.179:0/4216634740 <== mon.0 v1:172.21.15.84:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1640840851 0 0) 0x7efc880b1a60 con 0x7efc8809ac40 2024-03-29T11:19:33.087 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.096+0000 7efc6f7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:19:33.087 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.096+0000 7efc6f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:33.087 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.096+0000 7efc6f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:33.087 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.096+0000 7efc6f7fe640 10 monclient(hunting): my global_id is 4131 2024-03-29T11:19:33.087 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.096+0000 7efc6f7fe640 1 -- 172.21.15.179:0/4216634740 --> v1:172.21.15.84:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efc881d73f0 con 0x7efc8809ac40 2024-03-29T11:19:33.087 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.096+0000 7efc6f7fe640 1 -- 172.21.15.179:0/4216634740 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 535382104 0 0) 0x7efc881d8020 con 0x7efc880acd70 2024-03-29T11:19:33.087 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.096+0000 7efc6f7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:19:33.087 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.096+0000 7efc6f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:33.088 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.096+0000 7efc6f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:33.088 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.096+0000 7efc6f7fe640 10 monclient(hunting): my global_id is 4130 2024-03-29T11:19:33.088 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.096+0000 7efc6f7fe640 1 -- 172.21.15.179:0/4216634740 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efc880b1a60 con 0x7efc880acd70 2024-03-29T11:19:33.088 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.096+0000 7efc6f7fe640 1 -- 172.21.15.179:0/4216634740 <== mon.1 v1:172.21.15.179:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3679804937 0 0) 0x7efc74004eb0 con 0x7efc88139820 2024-03-29T11:19:33.088 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.097+0000 7efc6f7fe640 1 -- 172.21.15.179:0/4216634740 --> v1:172.21.15.179:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7efc881d8020 con 0x7efc88139820 2024-03-29T11:19:33.088 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.097+0000 7efc6f7fe640 1 -- 172.21.15.179:0/4216634740 <== mon.1 v1:172.21.15.179:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4236868694 0 0) 0x7efc80001c00 con 0x7efc88139820 2024-03-29T11:19:33.088 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.097+0000 7efc6f7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:33.088 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.097+0000 7efc6f7fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:19:33.088 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.097+0000 7efc6f7fe640 10 monclient(hunting): dump: 2024-03-29T11:19:33.089 INFO:teuthology.orchestra.run.smithi179.stderr:epoch 1 2024-03-29T11:19:33.089 INFO:teuthology.orchestra.run.smithi179.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:19:33.089 INFO:teuthology.orchestra.run.smithi179.stderr:last_changed 2024-03-29T11:18:53.864025+0000 2024-03-29T11:19:33.089 INFO:teuthology.orchestra.run.smithi179.stderr:created 2024-03-29T11:18:53.864025+0000 2024-03-29T11:19:33.089 INFO:teuthology.orchestra.run.smithi179.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:33.090 INFO:teuthology.orchestra.run.smithi179.stderr:election_strategy: 1 2024-03-29T11:19:33.090 INFO:teuthology.orchestra.run.smithi179.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T11:19:33.090 INFO:teuthology.orchestra.run.smithi179.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T11:19:33.090 INFO:teuthology.orchestra.run.smithi179.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T11:19:33.090 INFO:teuthology.orchestra.run.smithi179.stderr: 2024-03-29T11:19:33.090 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.097+0000 7efc6f7fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:19:33.090 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.097+0000 7efc6f7fe640 1 -- 172.21.15.179:0/4216634740 <== mon.0 v1:172.21.15.84:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3228532404 0 0) 0x7efc881d73f0 con 0x7efc8809ac40 2024-03-29T11:19:33.091 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.097+0000 7efc6f7fe640 1 -- 172.21.15.179:0/4216634740 --> v1:172.21.15.84:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7efc74004eb0 con 0x7efc8809ac40 2024-03-29T11:19:33.091 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.097+0000 7efc6f7fe640 1 -- 172.21.15.179:0/4216634740 <== mon.2 v1:172.21.15.84:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2843216537 0 0) 0x7efc880b1a60 con 0x7efc880acd70 2024-03-29T11:19:33.091 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.097+0000 7efc6f7fe640 1 -- 172.21.15.179:0/4216634740 --> v1:172.21.15.84:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7efc881d73f0 con 0x7efc880acd70 2024-03-29T11:19:33.091 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.097+0000 7efc6f7fe640 1 -- 172.21.15.179:0/4216634740 <== mon.0 v1:172.21.15.84:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4236868694 0 0) 0x7efc7c001b80 con 0x7efc8809ac40 2024-03-29T11:19:33.091 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.097+0000 7efc6f7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:33.091 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.097+0000 7efc6f7fe640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:19:33.091 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.097+0000 7efc6f7fe640 10 monclient(hunting): dump: 2024-03-29T11:19:33.091 INFO:teuthology.orchestra.run.smithi179.stderr:epoch 1 2024-03-29T11:19:33.092 INFO:teuthology.orchestra.run.smithi179.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:19:33.092 INFO:teuthology.orchestra.run.smithi179.stderr:last_changed 2024-03-29T11:18:53.864025+0000 2024-03-29T11:19:33.092 INFO:teuthology.orchestra.run.smithi179.stderr:created 2024-03-29T11:18:53.864025+0000 2024-03-29T11:19:33.092 INFO:teuthology.orchestra.run.smithi179.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:33.092 INFO:teuthology.orchestra.run.smithi179.stderr:election_strategy: 1 2024-03-29T11:19:33.092 INFO:teuthology.orchestra.run.smithi179.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T11:19:33.092 INFO:teuthology.orchestra.run.smithi179.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T11:19:33.092 INFO:teuthology.orchestra.run.smithi179.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T11:19:33.092 INFO:teuthology.orchestra.run.smithi179.stderr: 2024-03-29T11:19:33.092 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.097+0000 7efc6f7fe640 1 -- 172.21.15.179:0/4216634740 <== mon.2 v1:172.21.15.84:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4236868694 0 0) 0x7efc78001860 con 0x7efc880acd70 2024-03-29T11:19:33.093 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.097+0000 7efc6f7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:33.093 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.097+0000 7efc6f7fe640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:19:33.093 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.097+0000 7efc6f7fe640 10 monclient(hunting): dump: 2024-03-29T11:19:33.093 INFO:teuthology.orchestra.run.smithi179.stderr:epoch 1 2024-03-29T11:19:33.093 INFO:teuthology.orchestra.run.smithi179.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:19:33.093 INFO:teuthology.orchestra.run.smithi179.stderr:last_changed 2024-03-29T11:18:53.864025+0000 2024-03-29T11:19:33.093 INFO:teuthology.orchestra.run.smithi179.stderr:created 2024-03-29T11:18:53.864025+0000 2024-03-29T11:19:33.093 INFO:teuthology.orchestra.run.smithi179.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:33.093 INFO:teuthology.orchestra.run.smithi179.stderr:election_strategy: 1 2024-03-29T11:19:33.093 INFO:teuthology.orchestra.run.smithi179.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T11:19:33.093 INFO:teuthology.orchestra.run.smithi179.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T11:19:33.093 INFO:teuthology.orchestra.run.smithi179.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T11:19:33.093 INFO:teuthology.orchestra.run.smithi179.stderr: 2024-03-29T11:19:33.093 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.097+0000 7efc6f7fe640 1 -- 172.21.15.179:0/4216634740 <== mon.1 v1:172.21.15.179:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1626458053 0 0) 0x7efc881d8020 con 0x7efc88139820 2024-03-29T11:19:33.094 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.097+0000 7efc6f7fe640 1 -- 172.21.15.179:0/4216634740 >> v1:172.21.15.84:6790/0 conn(0x7efc880acd70 legacy=0x7efc88139110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:33.094 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.097+0000 7efc6f7fe640 1 -- 172.21.15.179:0/4216634740 >> v1:172.21.15.84:6789/0 conn(0x7efc8809ac40 legacy=0x7efc881d7910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:33.094 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.097+0000 7efc6f7fe640 10 monclient: _finish_hunting 0 2024-03-29T11:19:33.094 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.097+0000 7efc6f7fe640 1 monclient: found mon.b 2024-03-29T11:19:33.094 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.097+0000 7efc6f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:19:33.094 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.097+0000 7efc6f7fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-29T11:19:33.094 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.097+0000 7efc6f7fe640 1 -- 172.21.15.179:0/4216634740 --> v1:172.21.15.179:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc881d81b0 con 0x7efc88139820 2024-03-29T11:19:33.094 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.097+0000 7efc6f7fe640 10 monclient: _finish_auth 0 2024-03-29T11:19:33.094 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.097+0000 7efc6f7fe640 10 monclient: _check_auth_tickets 2024-03-29T11:19:33.094 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.097+0000 7efc6f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:19:33.094 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.097+0000 7efc8d3d6640 5 monclient: authenticate success, global_id 4114 2024-03-29T11:19:33.094 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.097+0000 7efc8d3d6640 10 monclient: _renew_subs 2024-03-29T11:19:33.094 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.097+0000 7efc8d3d6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-29T11:19:33.094 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.097+0000 7efc8d3d6640 1 -- 172.21.15.179:0/4216634740 --> v1:172.21.15.179:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efc880b07b0 con 0x7efc88139820 2024-03-29T11:19:33.095 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.097+0000 7efc8d3d6640 10 client.4114.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:19:33.095 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.097+0000 7efc8d3d6640 10 monclient: _renew_subs 2024-03-29T11:19:33.095 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.097+0000 7efc8d3d6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-29T11:19:33.095 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.097+0000 7efc8d3d6640 1 -- 172.21.15.179:0/4216634740 --> v1:172.21.15.179:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7efc880b0ce0 con 0x7efc88139820 2024-03-29T11:19:33.095 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.097+0000 7efc8d3d6640 1 librados: init done 2024-03-29T11:19:33.095 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.097+0000 7efc6f7fe640 1 -- 172.21.15.179:0/4216634740 <== mon.1 v1:172.21.15.179:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7efc800020d0 con 0x7efc88139820 2024-03-29T11:19:33.095 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.097+0000 7efc6f7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:19:33.095 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.097+0000 7efc6f7fe640 1 -- 172.21.15.179:0/4216634740 <== mon.1 v1:172.21.15.179:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4236868694 0 0) 0x7efc80003b30 con 0x7efc88139820 2024-03-29T11:19:33.095 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.097+0000 7efc6f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:33.095 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.097+0000 7efc6f7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:19:33.095 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.097+0000 7efc6f7fe640 10 monclient: dump: 2024-03-29T11:19:33.095 INFO:teuthology.orchestra.run.smithi179.stderr:epoch 1 2024-03-29T11:19:33.095 INFO:teuthology.orchestra.run.smithi179.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:19:33.095 INFO:teuthology.orchestra.run.smithi179.stderr:last_changed 2024-03-29T11:18:53.864025+0000 2024-03-29T11:19:33.095 INFO:teuthology.orchestra.run.smithi179.stderr:created 2024-03-29T11:18:53.864025+0000 2024-03-29T11:19:33.096 INFO:teuthology.orchestra.run.smithi179.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:33.096 INFO:teuthology.orchestra.run.smithi179.stderr:election_strategy: 1 2024-03-29T11:19:33.096 INFO:teuthology.orchestra.run.smithi179.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T11:19:33.096 INFO:teuthology.orchestra.run.smithi179.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T11:19:33.096 INFO:teuthology.orchestra.run.smithi179.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T11:19:33.096 INFO:teuthology.orchestra.run.smithi179.stderr: 2024-03-29T11:19:33.096 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.097+0000 7efc857fa640 4 set_mon_vals no callback set 2024-03-29T11:19:33.096 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.097+0000 7efc6f7fe640 1 -- 172.21.15.179:0/4216634740 <== mon.1 v1:172.21.15.179:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7efc80004020 con 0x7efc88139820 2024-03-29T11:19:33.096 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.097+0000 7efc6f7fe640 1 -- 172.21.15.179:0/4216634740 <== mon.1 v1:172.21.15.179:6789/0 8 ==== osd_map(4..4 src has 1..4) v4 ==== 955+0+0 (unknown 769653526 0 0) 0x7efc800042d0 con 0x7efc88139820 2024-03-29T11:19:33.096 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.097+0000 7efc6f7fe640 10 client.4114.objecter ms_dispatch 0x7efc88000b90 osd_map(4..4 src has 1..4) v4 2024-03-29T11:19:33.096 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.097+0000 7efc6f7fe640 3 client.4114.objecter handle_osd_map got epochs [4,4] > 0 2024-03-29T11:19:33.096 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.097+0000 7efc6f7fe640 3 client.4114.objecter handle_osd_map decoding full epoch 4 2024-03-29T11:19:33.096 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.097+0000 7efc6f7fe640 20 client.4114.objecter dump_active .. 0 homeless 2024-03-29T11:19:33.096 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.098+0000 7efc8d3d6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:33.096 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.098+0000 7efc8d3d6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:33.097 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.098+0000 7efc8d3d6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-29T11:19:33.097 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.098+0000 7efc8d3d6640 1 -- 172.21.15.179:0/4216634740 --> v1:172.21.15.179:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efc8809b020 con 0x7efc88139820 2024-03-29T11:19:33.097 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.100+0000 7efc6f7fe640 1 -- 172.21.15.179:0/4216634740 <== mon.1 v1:172.21.15.179:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7efc80004640 con 0x7efc88139820 2024-03-29T11:19:33.097 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.100+0000 7efc6f7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:33.097 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.100+0000 7efc6f7fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:19:33.102 INFO:tasks.ceph.mgr.y.smithi084.stderr:2024-03-29T11:19:33.125+0000 7f45957a8040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-29T11:19:33.149 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.173+0000 7efc8d3d6640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "11e60a5a-051d-453f-a190-66387cb35592", "id": 2}] 2024-03-29T11:19:33.150 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.173+0000 7efc8d3d6640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "11e60a5a-051d-453f-a190-66387cb35592", "id": 2}] 2024-03-29T11:19:33.150 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.173+0000 7efc8d3d6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-29T11:19:33.150 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.173+0000 7efc8d3d6640 1 -- 172.21.15.179:0/4216634740 --> v1:172.21.15.179:6789/0 -- mon_command({"prefix": "osd new", "uuid": "11e60a5a-051d-453f-a190-66387cb35592", "id": 2} v 0) v1 -- 0x7efc8804f730 con 0x7efc88139820 2024-03-29T11:19:33.156 INFO:tasks.ceph.mgr.y.smithi084.stderr:2024-03-29T11:19:33.180+0000 7f45957a8040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-29T11:19:33.157 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.181+0000 7efc6f7fe640 1 -- 172.21.15.179:0/4216634740 <== mon.1 v1:172.21.15.179:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "11e60a5a-051d-453f-a190-66387cb35592", "id": 2}]=0 v5) v1 ==== 112+0+2 (unknown 2155970462 0 216599125) 0x7efc80005b40 con 0x7efc88139820 2024-03-29T11:19:33.157 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.181+0000 7efc6f7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "11e60a5a-051d-453f-a190-66387cb35592", "id": 2}] 2024-03-29T11:19:33.157 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.181+0000 7efc6f7fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:19:33.158 INFO:teuthology.orchestra.run.smithi179.stdout:2 2024-03-29T11:19:33.159 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.181+0000 7efc8d3d6640 10 librados: watch_flush enter 2024-03-29T11:19:33.159 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.181+0000 7efc8d3d6640 10 librados: watch_flush exit 2024-03-29T11:19:33.159 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.181+0000 7efc8d3d6640 20 client.4114.objecter shutdown clearing up homeless session... 2024-03-29T11:19:33.159 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.181+0000 7efc8d3d6640 10 client.4114.objecter successfully canceled tick 2024-03-29T11:19:33.159 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.182+0000 7efc8d3d6640 10 monclient: shutdown 2024-03-29T11:19:33.159 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.182+0000 7efc8d3d6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:19:33.159 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.182+0000 7efc8d3d6640 1 -- 172.21.15.179:0/4216634740 >> v1:172.21.15.179:6789/0 conn(0x7efc88139820 legacy=0x7efc880b00a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:33.159 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.182+0000 7efc8d3d6640 1 -- 172.21.15.179:0/4216634740 shutdown_connections 2024-03-29T11:19:33.159 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.182+0000 7efc8d3d6640 1 -- 172.21.15.179:0/4216634740 >> 172.21.15.179:0/4216634740 conn(0x7efc88066b00 msgr2=0x7efc880b4a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:19:33.159 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.182+0000 7efc8d3d6640 1 -- 172.21.15.179:0/4216634740 shutdown_connections 2024-03-29T11:19:33.160 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.182+0000 7efc8d3d6640 1 -- 172.21.15.179:0/4216634740 wait complete. 2024-03-29T11:19:33.160 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.182+0000 7efc8d3d6640 1 librados: shutdown 2024-03-29T11:19:33.168 INFO:tasks.ceph.mgr.x.smithi179.stderr:2024-03-29T11:19:33.191+0000 7f8706237040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-29T11:19:33.169 DEBUG:teuthology.orchestra.run.smithi179:> sudo ceph --cluster ceph osd new 2c329fbe-bbc4-4030-bf87-d711cad04679 3 2024-03-29T11:19:33.201 INFO:tasks.ceph.mgr.y.smithi084.stderr:2024-03-29T11:19:33.225+0000 7f45957a8040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-29T11:19:33.225 INFO:tasks.ceph.mgr.x.smithi179.stderr:2024-03-29T11:19:33.249+0000 7f8706237040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-29T11:19:33.255 INFO:tasks.ceph.mgr.y.smithi084.stderr:2024-03-29T11:19:33.278+0000 7f45957a8040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-29T11:19:33.268 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.292+0000 7f4e97d0e640 10 monclient: get_monmap_and_config 2024-03-29T11:19:33.268 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.292+0000 7f4e97d0e640 10 monclient: build_initial_monmap 2024-03-29T11:19:33.268 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.292+0000 7f4e97d0e640 10 monclient: monmap: 2024-03-29T11:19:33.269 INFO:teuthology.orchestra.run.smithi179.stderr:epoch 0 2024-03-29T11:19:33.269 INFO:teuthology.orchestra.run.smithi179.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:19:33.269 INFO:teuthology.orchestra.run.smithi179.stderr:last_changed 2024-03-29T11:19:33.293048+0000 2024-03-29T11:19:33.269 INFO:teuthology.orchestra.run.smithi179.stderr:created 2024-03-29T11:19:33.293048+0000 2024-03-29T11:19:33.269 INFO:teuthology.orchestra.run.smithi179.stderr:min_mon_release 0 (unknown) 2024-03-29T11:19:33.269 INFO:teuthology.orchestra.run.smithi179.stderr:election_strategy: 1 2024-03-29T11:19:33.269 INFO:teuthology.orchestra.run.smithi179.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-29T11:19:33.269 INFO:teuthology.orchestra.run.smithi179.stderr:1: v1:172.21.15.179:6789/0 mon.noname-c 2024-03-29T11:19:33.269 INFO:teuthology.orchestra.run.smithi179.stderr:2: v1:172.21.15.84:6790/0 mon.noname-b 2024-03-29T11:19:33.269 INFO:teuthology.orchestra.run.smithi179.stderr: 2024-03-29T11:19:33.273 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.292+0000 7f4e97d0e640 1 Processor -- start 2024-03-29T11:19:33.273 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.292+0000 7f4e97d0e640 1 -- start start 2024-03-29T11:19:33.273 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.292+0000 7f4e97d0e640 10 monclient: init 2024-03-29T11:19:33.273 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.293+0000 7f4e97d0e640 10 monclient: _reopen_session rank -1 2024-03-29T11:19:33.273 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.293+0000 7f4e97d0e640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:19:33.273 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.293+0000 7f4e97d0e640 10 monclient(hunting): picked mon.noname-a con 0x7f4e9009ac40 addr v1:172.21.15.84:6789/0 2024-03-29T11:19:33.274 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.293+0000 7f4e97d0e640 10 monclient(hunting): picked mon.noname-c con 0x7f4e9009c870 addr v1:172.21.15.179:6789/0 2024-03-29T11:19:33.274 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.293+0000 7f4e97d0e640 10 monclient(hunting): picked mon.noname-b con 0x7f4e900b4360 addr v1:172.21.15.84:6790/0 2024-03-29T11:19:33.274 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.293+0000 7f4e97d0e640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4e9009dec0 con 0x7f4e9009ac40 2024-03-29T11:19:33.274 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.293+0000 7f4e97d0e640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4e9009da30 con 0x7f4e9009c870 2024-03-29T11:19:33.274 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.293+0000 7f4e97d0e640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4e900b39c0 con 0x7f4e900b4360 2024-03-29T11:19:33.274 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.293+0000 7f4e97d0e640 10 monclient(hunting): _renew_subs 2024-03-29T11:19:33.274 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.293+0000 7f4e97d0e640 10 monclient(hunting): authenticate will time out at 811.625607s 2024-03-29T11:19:33.274 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.293+0000 7f4e952a9640 1 --1- >> v1:172.21.15.179:6789/0 conn(0x7f4e9009c870 0x7f4e9013f3a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.179:6789/0 says I am v1:172.21.15.179:53438/0 (socket says 172.21.15.179:53438) 2024-03-29T11:19:33.274 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.293+0000 7f4e952a9640 1 -- 172.21.15.179:0/1828123030 learned_addr learned my addr 172.21.15.179:0/1828123030 (peer_addr_for_me v1:172.21.15.179:0/0) 2024-03-29T11:19:33.274 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.293+0000 7f4e94aa8640 1 -- 172.21.15.179:0/1828123030 <== mon.1 v1:172.21.15.179:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1790389577 0 0) 0x7f4e9009da30 con 0x7f4e9009c870 2024-03-29T11:19:33.274 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.293+0000 7f4e94aa8640 10 monclient(hunting): _negotiate 2024-03-29T11:19:33.274 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.293+0000 7f4e94aa8640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:33.274 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.293+0000 7f4e94aa8640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:33.274 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.293+0000 7f4e94aa8640 10 monclient(hunting): my global_id is 4117 2024-03-29T11:19:33.275 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.293+0000 7f4e94aa8640 1 -- 172.21.15.179:0/1828123030 --> v1:172.21.15.179:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4e7c003520 con 0x7f4e9009c870 2024-03-29T11:19:33.275 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.293+0000 7f4e94aa8640 1 -- 172.21.15.179:0/1828123030 <== mon.0 v1:172.21.15.84:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2244471079 0 0) 0x7f4e80001110 con 0x7f4e9009ac40 2024-03-29T11:19:33.275 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.293+0000 7f4e94aa8640 10 monclient(hunting): _negotiate 2024-03-29T11:19:33.275 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.293+0000 7f4e94aa8640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:33.275 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.293+0000 7f4e94aa8640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:33.275 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.293+0000 7f4e94aa8640 10 monclient(hunting): my global_id is 4134 2024-03-29T11:19:33.275 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.294+0000 7f4e94aa8640 1 -- 172.21.15.179:0/1828123030 --> v1:172.21.15.84:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4e9009da30 con 0x7f4e9009ac40 2024-03-29T11:19:33.275 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.294+0000 7f4e94aa8640 1 -- 172.21.15.179:0/1828123030 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 708600779 0 0) 0x7f4e900b39c0 con 0x7f4e900b4360 2024-03-29T11:19:33.275 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.294+0000 7f4e94aa8640 10 monclient(hunting): _negotiate 2024-03-29T11:19:33.275 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.294+0000 7f4e94aa8640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:33.275 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.294+0000 7f4e94aa8640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:33.275 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.294+0000 7f4e94aa8640 10 monclient(hunting): my global_id is 4133 2024-03-29T11:19:33.275 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.294+0000 7f4e94aa8640 1 -- 172.21.15.179:0/1828123030 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4e80001110 con 0x7f4e900b4360 2024-03-29T11:19:33.275 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.294+0000 7f4e94aa8640 1 -- 172.21.15.179:0/1828123030 <== mon.1 v1:172.21.15.179:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2708096388 0 0) 0x7f4e7c003520 con 0x7f4e9009c870 2024-03-29T11:19:33.276 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.294+0000 7f4e94aa8640 1 -- 172.21.15.179:0/1828123030 >> v1:172.21.15.84:6790/0 conn(0x7f4e900b4360 legacy=0x7f4e9014d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:33.276 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.294+0000 7f4e94aa8640 1 -- 172.21.15.179:0/1828123030 >> v1:172.21.15.84:6789/0 conn(0x7f4e9009ac40 legacy=0x7f4e900b4a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:33.276 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.294+0000 7f4e94aa8640 10 monclient: _finish_hunting 0 2024-03-29T11:19:33.276 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.294+0000 7f4e94aa8640 1 monclient: found mon.noname-c 2024-03-29T11:19:33.276 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.294+0000 7f4e94aa8640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:19:33.276 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.294+0000 7f4e94aa8640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.179:6789/0 2024-03-29T11:19:33.276 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.294+0000 7f4e94aa8640 1 -- 172.21.15.179:0/1828123030 --> v1:172.21.15.179:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e900b5160 con 0x7f4e9009c870 2024-03-29T11:19:33.276 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.294+0000 7f4e94aa8640 10 monclient: _finish_auth 0 2024-03-29T11:19:33.276 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.294+0000 7f4e94aa8640 10 monclient: _check_auth_tickets 2024-03-29T11:19:33.276 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.294+0000 7f4e94aa8640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:19:33.276 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.294+0000 7f4e94aa8640 1 -- 172.21.15.179:0/1828123030 <== mon.1 v1:172.21.15.179:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4236868694 0 0) 0x7f4e8c001690 con 0x7f4e9009c870 2024-03-29T11:19:33.276 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.294+0000 7f4e97d0e640 5 monclient: authenticate success, global_id 4117 2024-03-29T11:19:33.277 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.294+0000 7f4e97d0e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:19:33.277 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.294+0000 7f4e94aa8640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:33.277 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.294+0000 7f4e94aa8640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:19:33.277 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.294+0000 7f4e94aa8640 10 monclient: dump: 2024-03-29T11:19:33.277 INFO:teuthology.orchestra.run.smithi179.stderr:epoch 1 2024-03-29T11:19:33.277 INFO:teuthology.orchestra.run.smithi179.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:19:33.277 INFO:teuthology.orchestra.run.smithi179.stderr:last_changed 2024-03-29T11:18:53.864025+0000 2024-03-29T11:19:33.277 INFO:teuthology.orchestra.run.smithi179.stderr:created 2024-03-29T11:18:53.864025+0000 2024-03-29T11:19:33.277 INFO:teuthology.orchestra.run.smithi179.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:33.277 INFO:teuthology.orchestra.run.smithi179.stderr:election_strategy: 1 2024-03-29T11:19:33.277 INFO:teuthology.orchestra.run.smithi179.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T11:19:33.277 INFO:teuthology.orchestra.run.smithi179.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T11:19:33.277 INFO:teuthology.orchestra.run.smithi179.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T11:19:33.277 INFO:teuthology.orchestra.run.smithi179.stderr: 2024-03-29T11:19:33.278 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.294+0000 7f4e97d0e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:19:33.278 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.294+0000 7f4e94aa8640 1 -- 172.21.15.179:0/1828123030 <== mon.1 v1:172.21.15.179:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4e8c001870 con 0x7f4e9009c870 2024-03-29T11:19:33.278 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.294+0000 7f4e94aa8640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:19:33.278 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.294+0000 7f4e94aa8640 1 -- 172.21.15.179:0/1828123030 <== mon.1 v1:172.21.15.179:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4236868694 0 0) 0x7f4e8c001c80 con 0x7f4e9009c870 2024-03-29T11:19:33.278 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.294+0000 7f4e97d0e640 10 monclient: get_monmap_and_config success 2024-03-29T11:19:33.278 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.294+0000 7f4e97d0e640 4 set_mon_vals no callback set 2024-03-29T11:19:33.278 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.294+0000 7f4e94aa8640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:33.278 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.294+0000 7f4e94aa8640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:19:33.278 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.294+0000 7f4e94aa8640 10 monclient: dump: 2024-03-29T11:19:33.278 INFO:teuthology.orchestra.run.smithi179.stderr:epoch 1 2024-03-29T11:19:33.278 INFO:teuthology.orchestra.run.smithi179.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:19:33.278 INFO:teuthology.orchestra.run.smithi179.stderr:last_changed 2024-03-29T11:18:53.864025+0000 2024-03-29T11:19:33.278 INFO:teuthology.orchestra.run.smithi179.stderr:created 2024-03-29T11:18:53.864025+0000 2024-03-29T11:19:33.279 INFO:teuthology.orchestra.run.smithi179.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:33.279 INFO:teuthology.orchestra.run.smithi179.stderr:election_strategy: 1 2024-03-29T11:19:33.279 INFO:teuthology.orchestra.run.smithi179.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T11:19:33.279 INFO:teuthology.orchestra.run.smithi179.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T11:19:33.279 INFO:teuthology.orchestra.run.smithi179.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T11:19:33.279 INFO:teuthology.orchestra.run.smithi179.stderr: 2024-03-29T11:19:33.279 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.294+0000 7f4e97d0e640 10 monclient: shutdown 2024-03-29T11:19:33.279 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.294+0000 7f4e97d0e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:19:33.279 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.294+0000 7f4e97d0e640 1 -- 172.21.15.179:0/1828123030 >> v1:172.21.15.179:6789/0 conn(0x7f4e9009c870 legacy=0x7f4e9013f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:33.279 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.294+0000 7f4e97d0e640 1 -- 172.21.15.179:0/1828123030 shutdown_connections 2024-03-29T11:19:33.279 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.294+0000 7f4e97d0e640 1 -- 172.21.15.179:0/1828123030 >> 172.21.15.179:0/1828123030 conn(0x7f4e90066b00 msgr2=0x7f4e90066f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:19:33.279 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.294+0000 7f4e97d0e640 1 -- 172.21.15.179:0/1828123030 shutdown_connections 2024-03-29T11:19:33.279 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.294+0000 7f4e97d0e640 1 -- 172.21.15.179:0/1828123030 wait complete. 2024-03-29T11:19:33.279 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.295+0000 7f4e97d0e640 10 monclient: build_initial_monmap 2024-03-29T11:19:33.279 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.295+0000 7f4e97d0e640 10 monclient: monmap: 2024-03-29T11:19:33.279 INFO:teuthology.orchestra.run.smithi179.stderr:epoch 0 2024-03-29T11:19:33.280 INFO:teuthology.orchestra.run.smithi179.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:19:33.280 INFO:teuthology.orchestra.run.smithi179.stderr:last_changed 0.000000 2024-03-29T11:19:33.280 INFO:teuthology.orchestra.run.smithi179.stderr:created 0.000000 2024-03-29T11:19:33.280 INFO:teuthology.orchestra.run.smithi179.stderr:min_mon_release 0 (unknown) 2024-03-29T11:19:33.280 INFO:teuthology.orchestra.run.smithi179.stderr:election_strategy: 1 2024-03-29T11:19:33.280 INFO:teuthology.orchestra.run.smithi179.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-29T11:19:33.280 INFO:teuthology.orchestra.run.smithi179.stderr:1: v1:172.21.15.179:6789/0 mon.noname-b 2024-03-29T11:19:33.280 INFO:teuthology.orchestra.run.smithi179.stderr:2: v1:172.21.15.84:6790/0 mon.noname-c 2024-03-29T11:19:33.280 INFO:teuthology.orchestra.run.smithi179.stderr: 2024-03-29T11:19:33.280 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.295+0000 7f4e97d0e640 1 librados: starting msgr at 2024-03-29T11:19:33.280 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.295+0000 7f4e97d0e640 1 librados: starting objecter 2024-03-29T11:19:33.280 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.295+0000 7f4e97d0e640 1 Processor -- start 2024-03-29T11:19:33.280 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.295+0000 7f4e97d0e640 1 -- start start 2024-03-29T11:19:33.280 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.295+0000 7f4e97d0e640 1 librados: setting wanted keys 2024-03-29T11:19:33.280 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.295+0000 7f4e97d0e640 1 librados: calling monclient init 2024-03-29T11:19:33.280 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.295+0000 7f4e97d0e640 10 monclient: init 2024-03-29T11:19:33.281 INFO:tasks.ceph.mgr.x.smithi179.stderr:2024-03-29T11:19:33.296+0000 7f8706237040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-29T11:19:33.285 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.296+0000 7f4e97d0e640 10 monclient: _reopen_session rank -1 2024-03-29T11:19:33.285 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.296+0000 7f4e97d0e640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:19:33.285 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.296+0000 7f4e97d0e640 10 monclient(hunting): picked mon.noname-a con 0x7f4e9009ac40 addr v1:172.21.15.84:6789/0 2024-03-29T11:19:33.285 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.296+0000 7f4e97d0e640 10 monclient(hunting): picked mon.noname-c con 0x7f4e900b7eb0 addr v1:172.21.15.84:6790/0 2024-03-29T11:19:33.285 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.296+0000 7f4e97d0e640 10 monclient(hunting): picked mon.noname-b con 0x7f4e9013ffa0 addr v1:172.21.15.179:6789/0 2024-03-29T11:19:33.285 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.296+0000 7f4e97d0e640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4e90143060 con 0x7f4e9009ac40 2024-03-29T11:19:33.285 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.296+0000 7f4e97d0e640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4e901d7760 con 0x7f4e9013ffa0 2024-03-29T11:19:33.285 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.296+0000 7f4e97d0e640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4e901d78f0 con 0x7f4e900b7eb0 2024-03-29T11:19:33.285 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.296+0000 7f4e97d0e640 10 monclient(hunting): _renew_subs 2024-03-29T11:19:33.285 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.296+0000 7f4e97d0e640 10 monclient(hunting): authenticate will time out at 811.628592s 2024-03-29T11:19:33.286 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.296+0000 7f4e962ab640 1 --1- >> v1:172.21.15.179:6789/0 conn(0x7f4e9013ffa0 0x7f4e901e1a90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.179:6789/0 says I am v1:172.21.15.179:53442/0 (socket says 172.21.15.179:53442) 2024-03-29T11:19:33.286 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.296+0000 7f4e962ab640 1 -- 172.21.15.179:0/3401828324 learned_addr learned my addr 172.21.15.179:0/3401828324 (peer_addr_for_me v1:172.21.15.179:0/0) 2024-03-29T11:19:33.286 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.296+0000 7f4e867fc640 10 client.?.objecter ms_handle_connect 0x7f4e9013ffa0 2024-03-29T11:19:33.286 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.296+0000 7f4e867fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:19:33.286 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.296+0000 7f4e867fc640 10 client.?.objecter ms_handle_connect 0x7f4e9009ac40 2024-03-29T11:19:33.286 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.296+0000 7f4e867fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:19:33.286 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.296+0000 7f4e867fc640 10 client.?.objecter ms_handle_connect 0x7f4e900b7eb0 2024-03-29T11:19:33.286 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.296+0000 7f4e867fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:19:33.286 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.296+0000 7f4e867fc640 1 -- 172.21.15.179:0/3401828324 <== mon.1 v1:172.21.15.179:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3444706337 0 0) 0x7f4e901d7760 con 0x7f4e9013ffa0 2024-03-29T11:19:33.286 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.296+0000 7f4e867fc640 10 monclient(hunting): _negotiate 2024-03-29T11:19:33.286 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.296+0000 7f4e867fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:33.286 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.296+0000 7f4e867fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:33.286 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.296+0000 7f4e867fc640 10 monclient(hunting): my global_id is 4120 2024-03-29T11:19:33.287 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.296+0000 7f4e867fc640 1 -- 172.21.15.179:0/3401828324 --> v1:172.21.15.179:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4e74004eb0 con 0x7f4e9013ffa0 2024-03-29T11:19:33.287 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.296+0000 7f4e867fc640 1 -- 172.21.15.179:0/3401828324 <== mon.0 v1:172.21.15.84:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1548343360 0 0) 0x7f4e90143060 con 0x7f4e9009ac40 2024-03-29T11:19:33.287 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.296+0000 7f4e867fc640 10 monclient(hunting): _negotiate 2024-03-29T11:19:33.287 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.296+0000 7f4e867fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:33.287 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.296+0000 7f4e867fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:33.287 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.296+0000 7f4e867fc640 10 monclient(hunting): my global_id is 4137 2024-03-29T11:19:33.287 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.296+0000 7f4e867fc640 1 -- 172.21.15.179:0/3401828324 --> v1:172.21.15.84:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4e901d7760 con 0x7f4e9009ac40 2024-03-29T11:19:33.287 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.296+0000 7f4e867fc640 1 -- 172.21.15.179:0/3401828324 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3323170392 0 0) 0x7f4e901d78f0 con 0x7f4e900b7eb0 2024-03-29T11:19:33.287 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.296+0000 7f4e867fc640 10 monclient(hunting): _negotiate 2024-03-29T11:19:33.287 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.296+0000 7f4e867fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:33.287 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.296+0000 7f4e867fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:33.287 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.296+0000 7f4e867fc640 10 monclient(hunting): my global_id is 4136 2024-03-29T11:19:33.287 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.296+0000 7f4e867fc640 1 -- 172.21.15.179:0/3401828324 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4e90143060 con 0x7f4e900b7eb0 2024-03-29T11:19:33.287 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.296+0000 7f4e867fc640 1 -- 172.21.15.179:0/3401828324 <== mon.1 v1:172.21.15.179:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2882082639 0 0) 0x7f4e74004eb0 con 0x7f4e9013ffa0 2024-03-29T11:19:33.287 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.296+0000 7f4e867fc640 1 -- 172.21.15.179:0/3401828324 --> v1:172.21.15.179:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4e901d78f0 con 0x7f4e9013ffa0 2024-03-29T11:19:33.288 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.296+0000 7f4e867fc640 1 -- 172.21.15.179:0/3401828324 <== mon.1 v1:172.21.15.179:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4236868694 0 0) 0x7f4e88001c70 con 0x7f4e9013ffa0 2024-03-29T11:19:33.288 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.297+0000 7f4e867fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:33.288 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.297+0000 7f4e867fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:19:33.288 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.297+0000 7f4e867fc640 10 monclient(hunting): dump: 2024-03-29T11:19:33.288 INFO:teuthology.orchestra.run.smithi179.stderr:epoch 1 2024-03-29T11:19:33.288 INFO:teuthology.orchestra.run.smithi179.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:19:33.288 INFO:teuthology.orchestra.run.smithi179.stderr:last_changed 2024-03-29T11:18:53.864025+0000 2024-03-29T11:19:33.288 INFO:teuthology.orchestra.run.smithi179.stderr:created 2024-03-29T11:18:53.864025+0000 2024-03-29T11:19:33.288 INFO:teuthology.orchestra.run.smithi179.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:33.288 INFO:teuthology.orchestra.run.smithi179.stderr:election_strategy: 1 2024-03-29T11:19:33.288 INFO:teuthology.orchestra.run.smithi179.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T11:19:33.288 INFO:teuthology.orchestra.run.smithi179.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T11:19:33.288 INFO:teuthology.orchestra.run.smithi179.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T11:19:33.288 INFO:teuthology.orchestra.run.smithi179.stderr: 2024-03-29T11:19:33.289 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.297+0000 7f4e867fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:19:33.289 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.297+0000 7f4e867fc640 1 -- 172.21.15.179:0/3401828324 <== mon.0 v1:172.21.15.84:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4157875644 0 0) 0x7f4e901d7760 con 0x7f4e9009ac40 2024-03-29T11:19:33.289 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.297+0000 7f4e867fc640 1 -- 172.21.15.179:0/3401828324 --> v1:172.21.15.84:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4e74004eb0 con 0x7f4e9009ac40 2024-03-29T11:19:33.289 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.297+0000 7f4e867fc640 1 -- 172.21.15.179:0/3401828324 <== mon.0 v1:172.21.15.84:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4236868694 0 0) 0x7f4e80001920 con 0x7f4e9009ac40 2024-03-29T11:19:33.289 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.297+0000 7f4e867fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:33.289 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.297+0000 7f4e867fc640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:19:33.289 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.297+0000 7f4e867fc640 10 monclient(hunting): dump: 2024-03-29T11:19:33.289 INFO:teuthology.orchestra.run.smithi179.stderr:epoch 1 2024-03-29T11:19:33.289 INFO:teuthology.orchestra.run.smithi179.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:19:33.289 INFO:teuthology.orchestra.run.smithi179.stderr:last_changed 2024-03-29T11:18:53.864025+0000 2024-03-29T11:19:33.289 INFO:teuthology.orchestra.run.smithi179.stderr:created 2024-03-29T11:18:53.864025+0000 2024-03-29T11:19:33.289 INFO:teuthology.orchestra.run.smithi179.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:33.289 INFO:teuthology.orchestra.run.smithi179.stderr:election_strategy: 1 2024-03-29T11:19:33.289 INFO:teuthology.orchestra.run.smithi179.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T11:19:33.290 INFO:teuthology.orchestra.run.smithi179.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T11:19:33.290 INFO:teuthology.orchestra.run.smithi179.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T11:19:33.290 INFO:teuthology.orchestra.run.smithi179.stderr: 2024-03-29T11:19:33.290 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.297+0000 7f4e867fc640 1 -- 172.21.15.179:0/3401828324 <== mon.2 v1:172.21.15.84:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4277512935 0 0) 0x7f4e90143060 con 0x7f4e900b7eb0 2024-03-29T11:19:33.290 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.297+0000 7f4e867fc640 1 -- 172.21.15.179:0/3401828324 --> v1:172.21.15.84:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4e901d7760 con 0x7f4e900b7eb0 2024-03-29T11:19:33.290 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.297+0000 7f4e867fc640 1 -- 172.21.15.179:0/3401828324 <== mon.2 v1:172.21.15.84:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4236868694 0 0) 0x7f4e8c001b80 con 0x7f4e900b7eb0 2024-03-29T11:19:33.290 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.297+0000 7f4e867fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:33.290 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.297+0000 7f4e867fc640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:19:33.290 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.297+0000 7f4e867fc640 10 monclient(hunting): dump: 2024-03-29T11:19:33.290 INFO:teuthology.orchestra.run.smithi179.stderr:epoch 1 2024-03-29T11:19:33.290 INFO:teuthology.orchestra.run.smithi179.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:19:33.290 INFO:teuthology.orchestra.run.smithi179.stderr:last_changed 2024-03-29T11:18:53.864025+0000 2024-03-29T11:19:33.290 INFO:teuthology.orchestra.run.smithi179.stderr:created 2024-03-29T11:18:53.864025+0000 2024-03-29T11:19:33.290 INFO:teuthology.orchestra.run.smithi179.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:33.291 INFO:teuthology.orchestra.run.smithi179.stderr:election_strategy: 1 2024-03-29T11:19:33.291 INFO:teuthology.orchestra.run.smithi179.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T11:19:33.291 INFO:teuthology.orchestra.run.smithi179.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T11:19:33.291 INFO:teuthology.orchestra.run.smithi179.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T11:19:33.291 INFO:teuthology.orchestra.run.smithi179.stderr: 2024-03-29T11:19:33.291 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.297+0000 7f4e867fc640 1 -- 172.21.15.179:0/3401828324 <== mon.1 v1:172.21.15.179:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1628688214 0 0) 0x7f4e901d78f0 con 0x7f4e9013ffa0 2024-03-29T11:19:33.291 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.297+0000 7f4e867fc640 1 -- 172.21.15.179:0/3401828324 >> v1:172.21.15.84:6790/0 conn(0x7f4e900b7eb0 legacy=0x7f4e901d7050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:33.291 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.297+0000 7f4e867fc640 1 -- 172.21.15.179:0/3401828324 >> v1:172.21.15.84:6789/0 conn(0x7f4e9009ac40 legacy=0x7f4e901427d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:33.291 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.297+0000 7f4e867fc640 10 monclient: _finish_hunting 0 2024-03-29T11:19:33.291 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.297+0000 7f4e867fc640 1 monclient: found mon.b 2024-03-29T11:19:33.291 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.297+0000 7f4e867fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:19:33.291 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.297+0000 7f4e867fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-29T11:19:33.291 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.297+0000 7f4e867fc640 1 -- 172.21.15.179:0/3401828324 --> v1:172.21.15.179:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e90140460 con 0x7f4e9013ffa0 2024-03-29T11:19:33.291 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.297+0000 7f4e867fc640 10 monclient: _finish_auth 0 2024-03-29T11:19:33.291 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.297+0000 7f4e867fc640 10 monclient: _check_auth_tickets 2024-03-29T11:19:33.292 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.297+0000 7f4e867fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:19:33.292 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.297+0000 7f4e97d0e640 5 monclient: authenticate success, global_id 4120 2024-03-29T11:19:33.292 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.297+0000 7f4e97d0e640 10 monclient: _renew_subs 2024-03-29T11:19:33.292 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.297+0000 7f4e97d0e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-29T11:19:33.292 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.297+0000 7f4e97d0e640 1 -- 172.21.15.179:0/3401828324 --> v1:172.21.15.179:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4e90140710 con 0x7f4e9013ffa0 2024-03-29T11:19:33.292 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.297+0000 7f4e97d0e640 10 client.4120.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:19:33.292 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.297+0000 7f4e97d0e640 10 monclient: _renew_subs 2024-03-29T11:19:33.292 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.297+0000 7f4e97d0e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-29T11:19:33.292 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.297+0000 7f4e97d0e640 1 -- 172.21.15.179:0/3401828324 --> v1:172.21.15.179:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f4e901e3390 con 0x7f4e9013ffa0 2024-03-29T11:19:33.292 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.297+0000 7f4e97d0e640 1 librados: init done 2024-03-29T11:19:33.292 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.297+0000 7f4e867fc640 1 -- 172.21.15.179:0/3401828324 <== mon.1 v1:172.21.15.179:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4e88003990 con 0x7f4e9013ffa0 2024-03-29T11:19:33.292 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.297+0000 7f4e867fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:19:33.292 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.297+0000 7f4e867fc640 1 -- 172.21.15.179:0/3401828324 <== mon.1 v1:172.21.15.179:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4236868694 0 0) 0x7f4e88003d20 con 0x7f4e9013ffa0 2024-03-29T11:19:33.292 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.297+0000 7f4e867fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:33.292 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.297+0000 7f4e87fff640 4 set_mon_vals no callback set 2024-03-29T11:19:33.292 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.297+0000 7f4e867fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:19:33.293 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.297+0000 7f4e867fc640 10 monclient: dump: 2024-03-29T11:19:33.293 INFO:teuthology.orchestra.run.smithi179.stderr:epoch 1 2024-03-29T11:19:33.293 INFO:teuthology.orchestra.run.smithi179.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:19:33.293 INFO:teuthology.orchestra.run.smithi179.stderr:last_changed 2024-03-29T11:18:53.864025+0000 2024-03-29T11:19:33.293 INFO:teuthology.orchestra.run.smithi179.stderr:created 2024-03-29T11:18:53.864025+0000 2024-03-29T11:19:33.293 INFO:teuthology.orchestra.run.smithi179.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:33.293 INFO:teuthology.orchestra.run.smithi179.stderr:election_strategy: 1 2024-03-29T11:19:33.293 INFO:teuthology.orchestra.run.smithi179.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T11:19:33.293 INFO:teuthology.orchestra.run.smithi179.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T11:19:33.293 INFO:teuthology.orchestra.run.smithi179.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T11:19:33.293 INFO:teuthology.orchestra.run.smithi179.stderr: 2024-03-29T11:19:33.293 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.297+0000 7f4e867fc640 1 -- 172.21.15.179:0/3401828324 <== mon.1 v1:172.21.15.179:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f4e88004290 con 0x7f4e9013ffa0 2024-03-29T11:19:33.293 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.297+0000 7f4e867fc640 1 -- 172.21.15.179:0/3401828324 <== mon.1 v1:172.21.15.179:6789/0 8 ==== osd_map(5..5 src has 1..5) v4 ==== 1070+0+0 (unknown 2769119163 0 0) 0x7f4e88006a60 con 0x7f4e9013ffa0 2024-03-29T11:19:33.293 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.297+0000 7f4e867fc640 10 client.4120.objecter ms_dispatch 0x7f4e90000b90 osd_map(5..5 src has 1..5) v4 2024-03-29T11:19:33.293 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.297+0000 7f4e867fc640 3 client.4120.objecter handle_osd_map got epochs [5,5] > 0 2024-03-29T11:19:33.294 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.297+0000 7f4e867fc640 3 client.4120.objecter handle_osd_map decoding full epoch 5 2024-03-29T11:19:33.294 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.297+0000 7f4e867fc640 20 client.4120.objecter dump_active .. 0 homeless 2024-03-29T11:19:33.294 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.298+0000 7f4e97d0e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:33.294 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.298+0000 7f4e97d0e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:33.294 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.298+0000 7f4e97d0e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-29T11:19:33.294 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.298+0000 7f4e97d0e640 1 -- 172.21.15.179:0/3401828324 --> v1:172.21.15.179:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4e78005080 con 0x7f4e9013ffa0 2024-03-29T11:19:33.294 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.300+0000 7f4e867fc640 1 -- 172.21.15.179:0/3401828324 <== mon.1 v1:172.21.15.179:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f4e88004720 con 0x7f4e9013ffa0 2024-03-29T11:19:33.294 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.300+0000 7f4e867fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:33.294 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.300+0000 7f4e867fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:19:33.298 INFO:tasks.ceph.mgr.y.smithi084.stderr:2024-03-29T11:19:33.321+0000 7f45957a8040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-29T11:19:33.328 INFO:tasks.ceph.mgr.x.smithi179.stderr:2024-03-29T11:19:33.352+0000 7f8706237040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-29T11:19:33.349 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.372+0000 7f4e97d0e640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "2c329fbe-bbc4-4030-bf87-d711cad04679", "id": 3}] 2024-03-29T11:19:33.349 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.372+0000 7f4e97d0e640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "2c329fbe-bbc4-4030-bf87-d711cad04679", "id": 3}] 2024-03-29T11:19:33.349 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.372+0000 7f4e97d0e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-29T11:19:33.349 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.372+0000 7f4e97d0e640 1 -- 172.21.15.179:0/3401828324 --> v1:172.21.15.179:6789/0 -- mon_command({"prefix": "osd new", "uuid": "2c329fbe-bbc4-4030-bf87-d711cad04679", "id": 3} v 0) v1 -- 0x7f4e780058b0 con 0x7f4e9013ffa0 2024-03-29T11:19:33.353 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.376+0000 7f4e867fc640 1 -- 172.21.15.179:0/3401828324 <== mon.1 v1:172.21.15.179:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "2c329fbe-bbc4-4030-bf87-d711cad04679", "id": 3}]=0 v6) v1 ==== 112+0+2 (unknown 1682504566 0 525046306) 0x7f4e88006700 con 0x7f4e9013ffa0 2024-03-29T11:19:33.353 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.376+0000 7f4e867fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "2c329fbe-bbc4-4030-bf87-d711cad04679", "id": 3}] 2024-03-29T11:19:33.353 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.376+0000 7f4e867fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:19:33.353 INFO:teuthology.orchestra.run.smithi179.stdout:3 2024-03-29T11:19:33.355 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.377+0000 7f4e97d0e640 10 librados: watch_flush enter 2024-03-29T11:19:33.355 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.377+0000 7f4e97d0e640 10 librados: watch_flush exit 2024-03-29T11:19:33.355 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.377+0000 7f4e97d0e640 20 client.4120.objecter shutdown clearing up homeless session... 2024-03-29T11:19:33.355 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.377+0000 7f4e97d0e640 10 client.4120.objecter successfully canceled tick 2024-03-29T11:19:33.355 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.377+0000 7f4e97d0e640 10 monclient: shutdown 2024-03-29T11:19:33.355 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.377+0000 7f4e97d0e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:19:33.355 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.377+0000 7f4e97d0e640 1 -- 172.21.15.179:0/3401828324 >> v1:172.21.15.179:6789/0 conn(0x7f4e9013ffa0 legacy=0x7f4e901e1a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:33.355 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.377+0000 7f4e97d0e640 1 -- 172.21.15.179:0/3401828324 shutdown_connections 2024-03-29T11:19:33.355 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.377+0000 7f4e97d0e640 1 -- 172.21.15.179:0/3401828324 >> 172.21.15.179:0/3401828324 conn(0x7f4e90066b00 msgr2=0x7f4e9013d7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:19:33.355 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.377+0000 7f4e97d0e640 1 -- 172.21.15.179:0/3401828324 shutdown_connections 2024-03-29T11:19:33.355 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.377+0000 7f4e97d0e640 1 -- 172.21.15.179:0/3401828324 wait complete. 2024-03-29T11:19:33.355 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.377+0000 7f4e97d0e640 1 librados: shutdown 2024-03-29T11:19:33.364 DEBUG:teuthology.orchestra.run.smithi179:> sudo ceph --cluster ceph osd new f4213d4a-bd36-4433-91ae-d7e2b02481d3 4 2024-03-29T11:19:33.374 INFO:tasks.ceph.mgr.x.smithi179.stderr:2024-03-29T11:19:33.397+0000 7f8706237040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-29T11:19:33.413 INFO:tasks.ceph.mgr.y.smithi084.stderr:2024-03-29T11:19:33.436+0000 7f45957a8040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-29T11:19:33.465 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.488+0000 7f81b16d9640 10 monclient: get_monmap_and_config 2024-03-29T11:19:33.465 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.488+0000 7f81b16d9640 10 monclient: build_initial_monmap 2024-03-29T11:19:33.466 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.488+0000 7f81b16d9640 10 monclient: monmap: 2024-03-29T11:19:33.466 INFO:teuthology.orchestra.run.smithi179.stderr:epoch 0 2024-03-29T11:19:33.466 INFO:teuthology.orchestra.run.smithi179.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:19:33.466 INFO:teuthology.orchestra.run.smithi179.stderr:last_changed 2024-03-29T11:19:33.489433+0000 2024-03-29T11:19:33.466 INFO:teuthology.orchestra.run.smithi179.stderr:created 2024-03-29T11:19:33.489433+0000 2024-03-29T11:19:33.466 INFO:teuthology.orchestra.run.smithi179.stderr:min_mon_release 0 (unknown) 2024-03-29T11:19:33.466 INFO:teuthology.orchestra.run.smithi179.stderr:election_strategy: 1 2024-03-29T11:19:33.466 INFO:teuthology.orchestra.run.smithi179.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-29T11:19:33.466 INFO:teuthology.orchestra.run.smithi179.stderr:1: v1:172.21.15.179:6789/0 mon.noname-c 2024-03-29T11:19:33.467 INFO:teuthology.orchestra.run.smithi179.stderr:2: v1:172.21.15.84:6790/0 mon.noname-b 2024-03-29T11:19:33.467 INFO:teuthology.orchestra.run.smithi179.stderr: 2024-03-29T11:19:33.467 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.489+0000 7f81b16d9640 1 Processor -- start 2024-03-29T11:19:33.467 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.489+0000 7f81b16d9640 1 -- start start 2024-03-29T11:19:33.467 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.489+0000 7f81b16d9640 10 monclient: init 2024-03-29T11:19:33.479 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.489+0000 7f81b16d9640 10 monclient: _reopen_session rank -1 2024-03-29T11:19:33.479 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.489+0000 7f81b16d9640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:19:33.479 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.489+0000 7f81b16d9640 10 monclient(hunting): picked mon.noname-a con 0x7f81ac066920 addr v1:172.21.15.84:6789/0 2024-03-29T11:19:33.479 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.489+0000 7f81b16d9640 10 monclient(hunting): picked mon.noname-c con 0x7f81ac066d00 addr v1:172.21.15.179:6789/0 2024-03-29T11:19:33.479 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.489+0000 7f81b16d9640 10 monclient(hunting): picked mon.noname-b con 0x7f81ac0af280 addr v1:172.21.15.84:6790/0 2024-03-29T11:19:33.479 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.489+0000 7f81b16d9640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81ac0afd20 con 0x7f81ac066920 2024-03-29T11:19:33.479 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.489+0000 7f81b16d9640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81ac13fba0 con 0x7f81ac066d00 2024-03-29T11:19:33.479 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.489+0000 7f81b16d9640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81ac0b0f40 con 0x7f81ac0af280 2024-03-29T11:19:33.479 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.489+0000 7f81b16d9640 10 monclient(hunting): _renew_subs 2024-03-29T11:19:33.480 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.489+0000 7f81b16d9640 10 monclient(hunting): authenticate will time out at 811.821803s 2024-03-29T11:19:33.480 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.489+0000 7f81aa7fc640 1 --1- >> v1:172.21.15.179:6789/0 conn(0x7f81ac066d00 0x7f81ac072390 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.179:6789/0 says I am v1:172.21.15.179:53456/0 (socket says 172.21.15.179:53456) 2024-03-29T11:19:33.480 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.489+0000 7f81aa7fc640 1 -- 172.21.15.179:0/3372533685 learned_addr learned my addr 172.21.15.179:0/3372533685 (peer_addr_for_me v1:172.21.15.179:0/0) 2024-03-29T11:19:33.480 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.490+0000 7f81a9ffb640 1 -- 172.21.15.179:0/3372533685 <== mon.1 v1:172.21.15.179:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1870650646 0 0) 0x7f81ac13fba0 con 0x7f81ac066d00 2024-03-29T11:19:33.480 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.490+0000 7f81a9ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:19:33.480 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.490+0000 7f81a9ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:33.480 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.490+0000 7f81a9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:33.480 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.490+0000 7f81a9ffb640 10 monclient(hunting): my global_id is 4123 2024-03-29T11:19:33.480 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.490+0000 7f81a9ffb640 1 -- 172.21.15.179:0/3372533685 --> v1:172.21.15.179:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8194003520 con 0x7f81ac066d00 2024-03-29T11:19:33.481 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.490+0000 7f81a9ffb640 1 -- 172.21.15.179:0/3372533685 <== mon.0 v1:172.21.15.84:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2055426956 0 0) 0x7f81ac0afd20 con 0x7f81ac066920 2024-03-29T11:19:33.481 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.490+0000 7f81a9ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:19:33.481 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.490+0000 7f81a9ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:33.481 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.490+0000 7f81a9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:33.481 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.490+0000 7f81a9ffb640 10 monclient(hunting): my global_id is 4140 2024-03-29T11:19:33.481 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.490+0000 7f81a9ffb640 1 -- 172.21.15.179:0/3372533685 --> v1:172.21.15.84:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f81ac13fba0 con 0x7f81ac066920 2024-03-29T11:19:33.481 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.490+0000 7f81a9ffb640 1 -- 172.21.15.179:0/3372533685 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3200187663 0 0) 0x7f81ac0b0f40 con 0x7f81ac0af280 2024-03-29T11:19:33.481 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.490+0000 7f81a9ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:19:33.481 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.490+0000 7f81a9ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:33.481 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.490+0000 7f81a9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:33.482 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.490+0000 7f81a9ffb640 10 monclient(hunting): my global_id is 4139 2024-03-29T11:19:33.482 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.490+0000 7f81a9ffb640 1 -- 172.21.15.179:0/3372533685 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f81ac0afd20 con 0x7f81ac0af280 2024-03-29T11:19:33.482 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.490+0000 7f81a9ffb640 1 -- 172.21.15.179:0/3372533685 <== mon.1 v1:172.21.15.179:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3833213730 0 0) 0x7f8194003520 con 0x7f81ac066d00 2024-03-29T11:19:33.482 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.490+0000 7f81a9ffb640 1 -- 172.21.15.179:0/3372533685 >> v1:172.21.15.84:6790/0 conn(0x7f81ac0af280 legacy=0x7f81ac071750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:33.482 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.490+0000 7f81a9ffb640 1 -- 172.21.15.179:0/3372533685 >> v1:172.21.15.84:6789/0 conn(0x7f81ac066920 legacy=0x7f81ac145540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:33.482 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.490+0000 7f81a9ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:19:33.482 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.490+0000 7f81a9ffb640 1 monclient: found mon.noname-c 2024-03-29T11:19:33.482 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.490+0000 7f81a9ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:19:33.482 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.490+0000 7f81a9ffb640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.179:6789/0 2024-03-29T11:19:33.482 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.490+0000 7f81a9ffb640 1 -- 172.21.15.179:0/3372533685 --> v1:172.21.15.179:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81ac0b10d0 con 0x7f81ac066d00 2024-03-29T11:19:33.483 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.490+0000 7f81a9ffb640 10 monclient: _finish_auth 0 2024-03-29T11:19:33.483 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.490+0000 7f81a9ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:19:33.483 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.490+0000 7f81a9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:19:33.483 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.490+0000 7f81b16d9640 5 monclient: authenticate success, global_id 4123 2024-03-29T11:19:33.483 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.490+0000 7f81a9ffb640 1 -- 172.21.15.179:0/3372533685 <== mon.1 v1:172.21.15.179:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4236868694 0 0) 0x7f81a0001690 con 0x7f81ac066d00 2024-03-29T11:19:33.483 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.490+0000 7f81a9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:33.483 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.490+0000 7f81a9ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:19:33.483 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.490+0000 7f81a9ffb640 10 monclient: dump: 2024-03-29T11:19:33.483 INFO:teuthology.orchestra.run.smithi179.stderr:epoch 1 2024-03-29T11:19:33.483 INFO:teuthology.orchestra.run.smithi179.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:19:33.484 INFO:teuthology.orchestra.run.smithi179.stderr:last_changed 2024-03-29T11:18:53.864025+0000 2024-03-29T11:19:33.484 INFO:teuthology.orchestra.run.smithi179.stderr:created 2024-03-29T11:18:53.864025+0000 2024-03-29T11:19:33.484 INFO:teuthology.orchestra.run.smithi179.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:33.484 INFO:teuthology.orchestra.run.smithi179.stderr:election_strategy: 1 2024-03-29T11:19:33.484 INFO:teuthology.orchestra.run.smithi179.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T11:19:33.484 INFO:teuthology.orchestra.run.smithi179.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T11:19:33.484 INFO:teuthology.orchestra.run.smithi179.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T11:19:33.484 INFO:teuthology.orchestra.run.smithi179.stderr: 2024-03-29T11:19:33.484 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.490+0000 7f81b16d9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:19:33.484 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.490+0000 7f81a9ffb640 1 -- 172.21.15.179:0/3372533685 <== mon.1 v1:172.21.15.179:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f81a0001870 con 0x7f81ac066d00 2024-03-29T11:19:33.485 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.490+0000 7f81a9ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:19:33.485 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.490+0000 7f81b16d9640 10 monclient: get_monmap_and_config success 2024-03-29T11:19:33.485 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.490+0000 7f81a9ffb640 1 -- 172.21.15.179:0/3372533685 <== mon.1 v1:172.21.15.179:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4236868694 0 0) 0x7f81a0001c00 con 0x7f81ac066d00 2024-03-29T11:19:33.485 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.490+0000 7f81a9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:33.485 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.490+0000 7f81b16d9640 4 set_mon_vals no callback set 2024-03-29T11:19:33.485 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.490+0000 7f81a9ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:19:33.485 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.490+0000 7f81a9ffb640 10 monclient: dump: 2024-03-29T11:19:33.485 INFO:teuthology.orchestra.run.smithi179.stderr:epoch 1 2024-03-29T11:19:33.485 INFO:teuthology.orchestra.run.smithi179.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:19:33.485 INFO:teuthology.orchestra.run.smithi179.stderr:last_changed 2024-03-29T11:18:53.864025+0000 2024-03-29T11:19:33.486 INFO:teuthology.orchestra.run.smithi179.stderr:created 2024-03-29T11:18:53.864025+0000 2024-03-29T11:19:33.486 INFO:teuthology.orchestra.run.smithi179.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:33.486 INFO:teuthology.orchestra.run.smithi179.stderr:election_strategy: 1 2024-03-29T11:19:33.486 INFO:teuthology.orchestra.run.smithi179.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T11:19:33.486 INFO:teuthology.orchestra.run.smithi179.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T11:19:33.486 INFO:teuthology.orchestra.run.smithi179.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T11:19:33.486 INFO:teuthology.orchestra.run.smithi179.stderr: 2024-03-29T11:19:33.486 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.490+0000 7f81b16d9640 10 monclient: shutdown 2024-03-29T11:19:33.486 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.490+0000 7f81b16d9640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:19:33.487 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.490+0000 7f81b16d9640 1 -- 172.21.15.179:0/3372533685 >> v1:172.21.15.179:6789/0 conn(0x7f81ac066d00 legacy=0x7f81ac072390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:33.488 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.490+0000 7f81b16d9640 1 -- 172.21.15.179:0/3372533685 shutdown_connections 2024-03-29T11:19:33.488 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.490+0000 7f81b16d9640 1 -- 172.21.15.179:0/3372533685 >> 172.21.15.179:0/3372533685 conn(0x7f81ac0b4760 msgr2=0x7f81ac0b4b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:19:33.488 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.491+0000 7f81b16d9640 1 -- 172.21.15.179:0/3372533685 shutdown_connections 2024-03-29T11:19:33.488 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.491+0000 7f81b16d9640 1 -- 172.21.15.179:0/3372533685 wait complete. 2024-03-29T11:19:33.488 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.491+0000 7f81b16d9640 10 monclient: build_initial_monmap 2024-03-29T11:19:33.488 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.491+0000 7f81b16d9640 10 monclient: monmap: 2024-03-29T11:19:33.488 INFO:teuthology.orchestra.run.smithi179.stderr:epoch 0 2024-03-29T11:19:33.488 INFO:teuthology.orchestra.run.smithi179.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:19:33.488 INFO:teuthology.orchestra.run.smithi179.stderr:last_changed 0.000000 2024-03-29T11:19:33.488 INFO:teuthology.orchestra.run.smithi179.stderr:created 0.000000 2024-03-29T11:19:33.489 INFO:teuthology.orchestra.run.smithi179.stderr:min_mon_release 0 (unknown) 2024-03-29T11:19:33.489 INFO:teuthology.orchestra.run.smithi179.stderr:election_strategy: 1 2024-03-29T11:19:33.489 INFO:teuthology.orchestra.run.smithi179.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-29T11:19:33.489 INFO:teuthology.orchestra.run.smithi179.stderr:1: v1:172.21.15.179:6789/0 mon.noname-b 2024-03-29T11:19:33.489 INFO:teuthology.orchestra.run.smithi179.stderr:2: v1:172.21.15.84:6790/0 mon.noname-c 2024-03-29T11:19:33.489 INFO:teuthology.orchestra.run.smithi179.stderr: 2024-03-29T11:19:33.489 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.491+0000 7f81b16d9640 1 librados: starting msgr at 2024-03-29T11:19:33.489 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.491+0000 7f81b16d9640 1 librados: starting objecter 2024-03-29T11:19:33.489 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.492+0000 7f81b16d9640 1 Processor -- start 2024-03-29T11:19:33.490 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.492+0000 7f81b16d9640 1 -- start start 2024-03-29T11:19:33.490 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.492+0000 7f81b16d9640 1 librados: setting wanted keys 2024-03-29T11:19:33.490 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.492+0000 7f81b16d9640 1 librados: calling monclient init 2024-03-29T11:19:33.490 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.492+0000 7f81b16d9640 10 monclient: init 2024-03-29T11:19:33.490 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.492+0000 7f81b16d9640 10 monclient: _reopen_session rank -1 2024-03-29T11:19:33.490 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.492+0000 7f81b16d9640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:19:33.490 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.492+0000 7f81b16d9640 10 monclient(hunting): picked mon.noname-c con 0x7f81ac066920 addr v1:172.21.15.84:6790/0 2024-03-29T11:19:33.490 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.492+0000 7f81b16d9640 10 monclient(hunting): picked mon.noname-b con 0x7f81ac066d00 addr v1:172.21.15.179:6789/0 2024-03-29T11:19:33.490 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.492+0000 7f81b16d9640 10 monclient(hunting): picked mon.noname-a con 0x7f81ac0af280 addr v1:172.21.15.84:6789/0 2024-03-29T11:19:33.491 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.492+0000 7f81b16d9640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81ac14e6e0 con 0x7f81ac0af280 2024-03-29T11:19:33.491 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.492+0000 7f81b16d9640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81ac13d9f0 con 0x7f81ac066d00 2024-03-29T11:19:33.491 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.492+0000 7f81b16d9640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81ac13c870 con 0x7f81ac066920 2024-03-29T11:19:33.491 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.492+0000 7f81b16d9640 10 monclient(hunting): _renew_subs 2024-03-29T11:19:33.491 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.492+0000 7f81b16d9640 10 monclient(hunting): authenticate will time out at 811.824789s 2024-03-29T11:19:33.491 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.492+0000 7f81aa7fc640 1 --1- >> v1:172.21.15.179:6789/0 conn(0x7f81ac066d00 0x7f81ac138850 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.179:6789/0 says I am v1:172.21.15.179:53458/0 (socket says 172.21.15.179:53458) 2024-03-29T11:19:33.491 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.492+0000 7f81aa7fc640 1 -- 172.21.15.179:0/2864631237 learned_addr learned my addr 172.21.15.179:0/2864631237 (peer_addr_for_me v1:172.21.15.179:0/0) 2024-03-29T11:19:33.491 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.492+0000 7f81937fe640 10 client.?.objecter ms_handle_connect 0x7f81ac066d00 2024-03-29T11:19:33.491 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.492+0000 7f81937fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:19:33.492 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.492+0000 7f81937fe640 10 client.?.objecter ms_handle_connect 0x7f81ac0af280 2024-03-29T11:19:33.492 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.492+0000 7f81937fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:19:33.492 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.492+0000 7f81937fe640 10 client.?.objecter ms_handle_connect 0x7f81ac066920 2024-03-29T11:19:33.492 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.492+0000 7f81937fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:19:33.492 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.492+0000 7f81937fe640 1 -- 172.21.15.179:0/2864631237 <== mon.1 v1:172.21.15.179:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2521838510 0 0) 0x7f81ac13d9f0 con 0x7f81ac066d00 2024-03-29T11:19:33.492 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.492+0000 7f81937fe640 10 monclient(hunting): _negotiate 2024-03-29T11:19:33.492 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.492+0000 7f81937fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:33.492 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.492+0000 7f81937fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:33.492 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.492+0000 7f81937fe640 10 monclient(hunting): my global_id is 4126 2024-03-29T11:19:33.492 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.493+0000 7f81937fe640 1 -- 172.21.15.179:0/2864631237 --> v1:172.21.15.179:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8198004eb0 con 0x7f81ac066d00 2024-03-29T11:19:33.493 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.493+0000 7f81937fe640 1 -- 172.21.15.179:0/2864631237 <== mon.0 v1:172.21.15.84:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2150836696 0 0) 0x7f81ac14e6e0 con 0x7f81ac0af280 2024-03-29T11:19:33.493 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.493+0000 7f81937fe640 10 monclient(hunting): _negotiate 2024-03-29T11:19:33.493 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.493+0000 7f81937fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:33.493 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.493+0000 7f81937fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:33.493 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.493+0000 7f81937fe640 10 monclient(hunting): my global_id is 4143 2024-03-29T11:19:33.493 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.493+0000 7f81937fe640 1 -- 172.21.15.179:0/2864631237 --> v1:172.21.15.84:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f81ac13d9f0 con 0x7f81ac0af280 2024-03-29T11:19:33.493 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.493+0000 7f81937fe640 1 -- 172.21.15.179:0/2864631237 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 940975114 0 0) 0x7f81ac13c870 con 0x7f81ac066920 2024-03-29T11:19:33.493 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.493+0000 7f81937fe640 10 monclient(hunting): _negotiate 2024-03-29T11:19:33.493 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.493+0000 7f81937fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:33.493 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.493+0000 7f81937fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:33.493 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.493+0000 7f81937fe640 10 monclient(hunting): my global_id is 4142 2024-03-29T11:19:33.494 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.493+0000 7f81937fe640 1 -- 172.21.15.179:0/2864631237 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f81ac14e6e0 con 0x7f81ac066920 2024-03-29T11:19:33.494 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.493+0000 7f81937fe640 1 -- 172.21.15.179:0/2864631237 <== mon.1 v1:172.21.15.179:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3438244920 0 0) 0x7f8198004eb0 con 0x7f81ac066d00 2024-03-29T11:19:33.494 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.493+0000 7f81937fe640 1 -- 172.21.15.179:0/2864631237 --> v1:172.21.15.179:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f81ac13c870 con 0x7f81ac066d00 2024-03-29T11:19:33.494 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.493+0000 7f81937fe640 1 -- 172.21.15.179:0/2864631237 <== mon.1 v1:172.21.15.179:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4236868694 0 0) 0x7f81a0001b80 con 0x7f81ac066d00 2024-03-29T11:19:33.494 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.493+0000 7f81937fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:33.494 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.493+0000 7f81937fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:19:33.494 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.493+0000 7f81937fe640 10 monclient(hunting): dump: 2024-03-29T11:19:33.494 INFO:teuthology.orchestra.run.smithi179.stderr:epoch 1 2024-03-29T11:19:33.494 INFO:teuthology.orchestra.run.smithi179.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:19:33.494 INFO:teuthology.orchestra.run.smithi179.stderr:last_changed 2024-03-29T11:18:53.864025+0000 2024-03-29T11:19:33.495 INFO:teuthology.orchestra.run.smithi179.stderr:created 2024-03-29T11:18:53.864025+0000 2024-03-29T11:19:33.495 INFO:teuthology.orchestra.run.smithi179.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:33.495 INFO:teuthology.orchestra.run.smithi179.stderr:election_strategy: 1 2024-03-29T11:19:33.495 INFO:teuthology.orchestra.run.smithi179.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T11:19:33.495 INFO:teuthology.orchestra.run.smithi179.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T11:19:33.495 INFO:teuthology.orchestra.run.smithi179.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T11:19:33.495 INFO:teuthology.orchestra.run.smithi179.stderr: 2024-03-29T11:19:33.495 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.493+0000 7f81937fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:19:33.495 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.493+0000 7f81937fe640 1 -- 172.21.15.179:0/2864631237 <== mon.0 v1:172.21.15.84:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3680986102 0 0) 0x7f81ac13d9f0 con 0x7f81ac0af280 2024-03-29T11:19:33.495 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.493+0000 7f81937fe640 1 -- 172.21.15.179:0/2864631237 --> v1:172.21.15.84:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8198004eb0 con 0x7f81ac0af280 2024-03-29T11:19:33.496 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.493+0000 7f81937fe640 1 -- 172.21.15.179:0/2864631237 <== mon.0 v1:172.21.15.84:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4236868694 0 0) 0x7f81a4001a40 con 0x7f81ac0af280 2024-03-29T11:19:33.496 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.493+0000 7f81937fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:33.496 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.493+0000 7f81937fe640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:19:33.496 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.493+0000 7f81937fe640 10 monclient(hunting): dump: 2024-03-29T11:19:33.496 INFO:teuthology.orchestra.run.smithi179.stderr:epoch 1 2024-03-29T11:19:33.496 INFO:teuthology.orchestra.run.smithi179.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:19:33.496 INFO:teuthology.orchestra.run.smithi179.stderr:last_changed 2024-03-29T11:18:53.864025+0000 2024-03-29T11:19:33.496 INFO:teuthology.orchestra.run.smithi179.stderr:created 2024-03-29T11:18:53.864025+0000 2024-03-29T11:19:33.496 INFO:teuthology.orchestra.run.smithi179.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:33.497 INFO:teuthology.orchestra.run.smithi179.stderr:election_strategy: 1 2024-03-29T11:19:33.497 INFO:teuthology.orchestra.run.smithi179.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T11:19:33.497 INFO:teuthology.orchestra.run.smithi179.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T11:19:33.497 INFO:teuthology.orchestra.run.smithi179.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T11:19:33.497 INFO:teuthology.orchestra.run.smithi179.stderr: 2024-03-29T11:19:33.497 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.493+0000 7f81937fe640 1 -- 172.21.15.179:0/2864631237 <== mon.2 v1:172.21.15.84:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 459634848 0 0) 0x7f81ac14e6e0 con 0x7f81ac066920 2024-03-29T11:19:33.497 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.493+0000 7f81937fe640 1 -- 172.21.15.179:0/2864631237 --> v1:172.21.15.84:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f81ac13d9f0 con 0x7f81ac066920 2024-03-29T11:19:33.497 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.493+0000 7f81937fe640 1 -- 172.21.15.179:0/2864631237 <== mon.2 v1:172.21.15.84:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4236868694 0 0) 0x7f819c0018b0 con 0x7f81ac066920 2024-03-29T11:19:33.497 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.493+0000 7f81937fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:33.498 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.493+0000 7f81937fe640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:19:33.498 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.493+0000 7f81937fe640 10 monclient(hunting): dump: 2024-03-29T11:19:33.498 INFO:teuthology.orchestra.run.smithi179.stderr:epoch 1 2024-03-29T11:19:33.498 INFO:teuthology.orchestra.run.smithi179.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:19:33.498 INFO:teuthology.orchestra.run.smithi179.stderr:last_changed 2024-03-29T11:18:53.864025+0000 2024-03-29T11:19:33.498 INFO:teuthology.orchestra.run.smithi179.stderr:created 2024-03-29T11:18:53.864025+0000 2024-03-29T11:19:33.498 INFO:teuthology.orchestra.run.smithi179.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:33.498 INFO:teuthology.orchestra.run.smithi179.stderr:election_strategy: 1 2024-03-29T11:19:33.498 INFO:teuthology.orchestra.run.smithi179.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T11:19:33.499 INFO:teuthology.orchestra.run.smithi179.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T11:19:33.499 INFO:teuthology.orchestra.run.smithi179.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T11:19:33.499 INFO:teuthology.orchestra.run.smithi179.stderr: 2024-03-29T11:19:33.499 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.493+0000 7f81937fe640 1 -- 172.21.15.179:0/2864631237 <== mon.1 v1:172.21.15.179:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2499946222 0 0) 0x7f81ac13c870 con 0x7f81ac066d00 2024-03-29T11:19:33.499 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.493+0000 7f81937fe640 1 -- 172.21.15.179:0/2864631237 >> v1:172.21.15.84:6790/0 conn(0x7f81ac066920 legacy=0x7f81ac13c160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:33.499 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.493+0000 7f81937fe640 1 -- 172.21.15.179:0/2864631237 >> v1:172.21.15.84:6789/0 conn(0x7f81ac0af280 legacy=0x7f81ac1e26e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:33.499 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.493+0000 7f81937fe640 10 monclient: _finish_hunting 0 2024-03-29T11:19:33.499 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.493+0000 7f81937fe640 1 monclient: found mon.b 2024-03-29T11:19:33.499 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.493+0000 7f81937fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:19:33.500 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.493+0000 7f81937fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-29T11:19:33.500 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.493+0000 7f81937fe640 1 -- 172.21.15.179:0/2864631237 --> v1:172.21.15.179:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81ac13cb90 con 0x7f81ac066d00 2024-03-29T11:19:33.500 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.493+0000 7f81937fe640 10 monclient: _finish_auth 0 2024-03-29T11:19:33.500 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.493+0000 7f81937fe640 10 monclient: _check_auth_tickets 2024-03-29T11:19:33.500 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.493+0000 7f81937fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:19:33.500 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.493+0000 7f81b16d9640 5 monclient: authenticate success, global_id 4126 2024-03-29T11:19:33.500 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.493+0000 7f81b16d9640 10 monclient: _renew_subs 2024-03-29T11:19:33.500 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.493+0000 7f81b16d9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-29T11:19:33.501 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.493+0000 7f81b16d9640 1 -- 172.21.15.179:0/2864631237 --> v1:172.21.15.179:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f81ac13cde0 con 0x7f81ac066d00 2024-03-29T11:19:33.501 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.493+0000 7f81b16d9640 10 client.4126.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:19:33.501 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.493+0000 7f81b16d9640 10 monclient: _renew_subs 2024-03-29T11:19:33.501 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.493+0000 7f81b16d9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-29T11:19:33.501 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.493+0000 7f81b16d9640 1 -- 172.21.15.179:0/2864631237 --> v1:172.21.15.179:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f81ac14ddb0 con 0x7f81ac066d00 2024-03-29T11:19:33.501 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.493+0000 7f81b16d9640 1 librados: init done 2024-03-29T11:19:33.501 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.493+0000 7f81937fe640 1 -- 172.21.15.179:0/2864631237 <== mon.1 v1:172.21.15.179:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f81a0004a90 con 0x7f81ac066d00 2024-03-29T11:19:33.501 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.493+0000 7f81937fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:19:33.501 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.493+0000 7f81937fe640 1 -- 172.21.15.179:0/2864631237 <== mon.1 v1:172.21.15.179:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4236868694 0 0) 0x7f81a0004da0 con 0x7f81ac066d00 2024-03-29T11:19:33.501 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.493+0000 7f81937fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:33.502 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.493+0000 7f81937fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:19:33.502 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.493+0000 7f81a97fa640 4 set_mon_vals no callback set 2024-03-29T11:19:33.502 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.493+0000 7f81937fe640 10 monclient: dump: 2024-03-29T11:19:33.502 INFO:teuthology.orchestra.run.smithi179.stderr:epoch 1 2024-03-29T11:19:33.502 INFO:teuthology.orchestra.run.smithi179.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:19:33.502 INFO:teuthology.orchestra.run.smithi179.stderr:last_changed 2024-03-29T11:18:53.864025+0000 2024-03-29T11:19:33.502 INFO:teuthology.orchestra.run.smithi179.stderr:created 2024-03-29T11:18:53.864025+0000 2024-03-29T11:19:33.502 INFO:teuthology.orchestra.run.smithi179.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:33.502 INFO:teuthology.orchestra.run.smithi179.stderr:election_strategy: 1 2024-03-29T11:19:33.503 INFO:teuthology.orchestra.run.smithi179.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T11:19:33.503 INFO:teuthology.orchestra.run.smithi179.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T11:19:33.503 INFO:teuthology.orchestra.run.smithi179.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T11:19:33.503 INFO:teuthology.orchestra.run.smithi179.stderr: 2024-03-29T11:19:33.503 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.493+0000 7f81937fe640 1 -- 172.21.15.179:0/2864631237 <== mon.1 v1:172.21.15.179:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f81a0005350 con 0x7f81ac066d00 2024-03-29T11:19:33.503 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.493+0000 7f81937fe640 1 -- 172.21.15.179:0/2864631237 <== mon.1 v1:172.21.15.179:6789/0 8 ==== osd_map(6..6 src has 1..6) v4 ==== 1185+0+0 (unknown 2074209873 0 0) 0x7f81a0006500 con 0x7f81ac066d00 2024-03-29T11:19:33.503 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.493+0000 7f81937fe640 10 client.4126.objecter ms_dispatch 0x7f81ac000b90 osd_map(6..6 src has 1..6) v4 2024-03-29T11:19:33.503 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.493+0000 7f81937fe640 3 client.4126.objecter handle_osd_map got epochs [6,6] > 0 2024-03-29T11:19:33.503 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.493+0000 7f81937fe640 3 client.4126.objecter handle_osd_map decoding full epoch 6 2024-03-29T11:19:33.504 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.493+0000 7f81937fe640 20 client.4126.objecter dump_active .. 0 homeless 2024-03-29T11:19:33.504 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.494+0000 7f81b16d9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:33.504 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.494+0000 7f81b16d9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:33.504 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.494+0000 7f81b16d9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-29T11:19:33.504 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.494+0000 7f81b16d9640 1 -- 172.21.15.179:0/2864631237 --> v1:172.21.15.179:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f818c005080 con 0x7f81ac066d00 2024-03-29T11:19:33.504 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.496+0000 7f81937fe640 1 -- 172.21.15.179:0/2864631237 <== mon.1 v1:172.21.15.179:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f81a0006790 con 0x7f81ac066d00 2024-03-29T11:19:33.504 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.496+0000 7f81937fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:33.504 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.496+0000 7f81937fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:19:33.505 INFO:tasks.ceph.mgr.x.smithi179.stderr:2024-03-29T11:19:33.520+0000 7f8706237040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-29T11:19:33.506 INFO:tasks.ceph.mgr.y.smithi084.stderr:2024-03-29T11:19:33.505+0000 7f45957a8040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-29T11:19:33.528 INFO:tasks.ceph.mgr.y.smithi084.stderr:2024-03-29T11:19:33.552+0000 7f45957a8040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-29T11:19:33.545 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.568+0000 7f81b16d9640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "f4213d4a-bd36-4433-91ae-d7e2b02481d3", "id": 4}] 2024-03-29T11:19:33.545 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.568+0000 7f81b16d9640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "f4213d4a-bd36-4433-91ae-d7e2b02481d3", "id": 4}] 2024-03-29T11:19:33.545 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.568+0000 7f81b16d9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-29T11:19:33.545 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.568+0000 7f81b16d9640 1 -- 172.21.15.179:0/2864631237 --> v1:172.21.15.179:6789/0 -- mon_command({"prefix": "osd new", "uuid": "f4213d4a-bd36-4433-91ae-d7e2b02481d3", "id": 4} v 0) v1 -- 0x7f818c0058b0 con 0x7f81ac066d00 2024-03-29T11:19:33.549 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.572+0000 7f81937fe640 1 -- 172.21.15.179:0/2864631237 <== mon.1 v1:172.21.15.179:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "f4213d4a-bd36-4433-91ae-d7e2b02481d3", "id": 4}]=0 v7) v1 ==== 112+0+2 (unknown 3195755537 0 1697012583) 0x7f81a00069f0 con 0x7f81ac066d00 2024-03-29T11:19:33.549 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.572+0000 7f81937fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "f4213d4a-bd36-4433-91ae-d7e2b02481d3", "id": 4}] 2024-03-29T11:19:33.549 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.572+0000 7f81937fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:19:33.549 INFO:teuthology.orchestra.run.smithi179.stdout:4 2024-03-29T11:19:33.550 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.573+0000 7f81b16d9640 10 librados: watch_flush enter 2024-03-29T11:19:33.551 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.573+0000 7f81b16d9640 10 librados: watch_flush exit 2024-03-29T11:19:33.551 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.573+0000 7f81b16d9640 20 client.4126.objecter shutdown clearing up homeless session... 2024-03-29T11:19:33.551 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.573+0000 7f81b16d9640 10 client.4126.objecter successfully canceled tick 2024-03-29T11:19:33.551 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.573+0000 7f81b16d9640 10 monclient: shutdown 2024-03-29T11:19:33.551 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.573+0000 7f81b16d9640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:19:33.551 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.573+0000 7f81b16d9640 1 -- 172.21.15.179:0/2864631237 >> v1:172.21.15.179:6789/0 conn(0x7f81ac066d00 legacy=0x7f81ac138850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:33.551 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.573+0000 7f81b16d9640 1 -- 172.21.15.179:0/2864631237 shutdown_connections 2024-03-29T11:19:33.551 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.573+0000 7f81b16d9640 1 -- 172.21.15.179:0/2864631237 >> 172.21.15.179:0/2864631237 conn(0x7f81ac0b4760 msgr2=0x7f81ac069bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:19:33.551 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.573+0000 7f81b16d9640 1 -- 172.21.15.179:0/2864631237 shutdown_connections 2024-03-29T11:19:33.551 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.573+0000 7f81b16d9640 1 -- 172.21.15.179:0/2864631237 wait complete. 2024-03-29T11:19:33.551 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.573+0000 7f81b16d9640 1 librados: shutdown 2024-03-29T11:19:33.560 DEBUG:teuthology.orchestra.run.smithi179:> sudo ceph --cluster ceph osd new f0cf44ea-6055-48af-805b-aed858102ff9 5 2024-03-29T11:19:33.570 INFO:tasks.ceph.mgr.x.smithi179.stderr:2024-03-29T11:19:33.594+0000 7f8706237040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-29T11:19:33.582 INFO:tasks.ceph.mgr.y.smithi084.stderr:2024-03-29T11:19:33.605+0000 7f45957a8040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-29T11:19:33.619 INFO:tasks.ceph.mgr.x.smithi179.stderr:2024-03-29T11:19:33.642+0000 7f8706237040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-29T11:19:33.628 INFO:tasks.ceph.mgr.y.smithi084.stderr:2024-03-29T11:19:33.651+0000 7f45957a8040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-29T11:19:33.669 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.693+0000 7f46a10e0640 10 monclient: get_monmap_and_config 2024-03-29T11:19:33.670 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.693+0000 7f46a10e0640 10 monclient: build_initial_monmap 2024-03-29T11:19:33.670 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.693+0000 7f46a10e0640 10 monclient: monmap: 2024-03-29T11:19:33.670 INFO:teuthology.orchestra.run.smithi179.stderr:epoch 0 2024-03-29T11:19:33.670 INFO:teuthology.orchestra.run.smithi179.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:19:33.670 INFO:teuthology.orchestra.run.smithi179.stderr:last_changed 2024-03-29T11:19:33.694133+0000 2024-03-29T11:19:33.670 INFO:teuthology.orchestra.run.smithi179.stderr:created 2024-03-29T11:19:33.694133+0000 2024-03-29T11:19:33.670 INFO:teuthology.orchestra.run.smithi179.stderr:min_mon_release 0 (unknown) 2024-03-29T11:19:33.670 INFO:teuthology.orchestra.run.smithi179.stderr:election_strategy: 1 2024-03-29T11:19:33.670 INFO:teuthology.orchestra.run.smithi179.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-29T11:19:33.670 INFO:teuthology.orchestra.run.smithi179.stderr:1: v1:172.21.15.179:6789/0 mon.noname-c 2024-03-29T11:19:33.670 INFO:teuthology.orchestra.run.smithi179.stderr:2: v1:172.21.15.84:6790/0 mon.noname-b 2024-03-29T11:19:33.670 INFO:teuthology.orchestra.run.smithi179.stderr: 2024-03-29T11:19:33.678 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.693+0000 7f46a10e0640 1 Processor -- start 2024-03-29T11:19:33.678 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.693+0000 7f46a10e0640 1 -- start start 2024-03-29T11:19:33.678 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.693+0000 7f46a10e0640 10 monclient: init 2024-03-29T11:19:33.678 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.694+0000 7f46a10e0640 10 monclient: _reopen_session rank -1 2024-03-29T11:19:33.678 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.694+0000 7f46a10e0640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:19:33.678 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.694+0000 7f46a10e0640 10 monclient(hunting): picked mon.noname-c con 0x7f469c0b1780 addr v1:172.21.15.179:6789/0 2024-03-29T11:19:33.678 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.694+0000 7f46a10e0640 10 monclient(hunting): picked mon.noname-a con 0x7f469c13d7e0 addr v1:172.21.15.84:6789/0 2024-03-29T11:19:33.678 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.694+0000 7f46a10e0640 10 monclient(hunting): picked mon.noname-b con 0x7f469c143be0 addr v1:172.21.15.84:6790/0 2024-03-29T11:19:33.678 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.694+0000 7f46a10e0640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f469c070b30 con 0x7f469c13d7e0 2024-03-29T11:19:33.678 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.694+0000 7f46a10e0640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f469c070da0 con 0x7f469c0b1780 2024-03-29T11:19:33.679 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.694+0000 7f46a10e0640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f469c09adb0 con 0x7f469c143be0 2024-03-29T11:19:33.679 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.694+0000 7f46a10e0640 10 monclient(hunting): _renew_subs 2024-03-29T11:19:33.679 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.694+0000 7f46a10e0640 10 monclient(hunting): authenticate will time out at 812.026596s 2024-03-29T11:19:33.679 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.694+0000 7f469ad9d640 1 --1- >> v1:172.21.15.179:6789/0 conn(0x7f469c0b1780 0x7f469c13d0d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.179:6789/0 says I am v1:172.21.15.179:53462/0 (socket says 172.21.15.179:53462) 2024-03-29T11:19:33.679 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.694+0000 7f469ad9d640 1 -- 172.21.15.179:0/3882078564 learned_addr learned my addr 172.21.15.179:0/3882078564 (peer_addr_for_me v1:172.21.15.179:0/0) 2024-03-29T11:19:33.679 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.694+0000 7f4699d9b640 1 -- 172.21.15.179:0/3882078564 <== mon.1 v1:172.21.15.179:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4271531058 0 0) 0x7f469c070da0 con 0x7f469c0b1780 2024-03-29T11:19:33.679 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.694+0000 7f4699d9b640 10 monclient(hunting): _negotiate 2024-03-29T11:19:33.679 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.694+0000 7f4699d9b640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:33.679 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.694+0000 7f4699d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:33.679 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.694+0000 7f4699d9b640 10 monclient(hunting): my global_id is 4129 2024-03-29T11:19:33.679 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.694+0000 7f4699d9b640 1 -- 172.21.15.179:0/3882078564 --> v1:172.21.15.179:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4684003520 con 0x7f469c0b1780 2024-03-29T11:19:33.679 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.694+0000 7f4699d9b640 1 -- 172.21.15.179:0/3882078564 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 717193576 0 0) 0x7f469c09adb0 con 0x7f469c143be0 2024-03-29T11:19:33.679 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.695+0000 7f4699d9b640 10 monclient(hunting): _negotiate 2024-03-29T11:19:33.679 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.695+0000 7f4699d9b640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:33.679 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.695+0000 7f4699d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:33.680 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.695+0000 7f4699d9b640 10 monclient(hunting): my global_id is 4145 2024-03-29T11:19:33.680 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.695+0000 7f4699d9b640 1 -- 172.21.15.179:0/3882078564 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f469c070da0 con 0x7f469c143be0 2024-03-29T11:19:33.680 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.695+0000 7f4699d9b640 1 -- 172.21.15.179:0/3882078564 <== mon.0 v1:172.21.15.84:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3059823307 0 0) 0x7f469c070b30 con 0x7f469c13d7e0 2024-03-29T11:19:33.680 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.695+0000 7f4699d9b640 10 monclient(hunting): _negotiate 2024-03-29T11:19:33.680 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.695+0000 7f4699d9b640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:33.680 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.695+0000 7f4699d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:33.680 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.695+0000 7f4699d9b640 10 monclient(hunting): my global_id is 4146 2024-03-29T11:19:33.680 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.695+0000 7f4699d9b640 1 -- 172.21.15.179:0/3882078564 --> v1:172.21.15.84:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f469c09adb0 con 0x7f469c13d7e0 2024-03-29T11:19:33.680 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.695+0000 7f4699d9b640 1 -- 172.21.15.179:0/3882078564 <== mon.1 v1:172.21.15.179:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1950098663 0 0) 0x7f4684003520 con 0x7f469c0b1780 2024-03-29T11:19:33.680 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.695+0000 7f4699d9b640 1 -- 172.21.15.179:0/3882078564 >> v1:172.21.15.84:6790/0 conn(0x7f469c143be0 legacy=0x7f469c14b780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:33.680 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.695+0000 7f4699d9b640 1 -- 172.21.15.179:0/3882078564 >> v1:172.21.15.84:6789/0 conn(0x7f469c13d7e0 legacy=0x7f469c1434d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:33.680 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.695+0000 7f4699d9b640 10 monclient: _finish_hunting 0 2024-03-29T11:19:33.680 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.695+0000 7f4699d9b640 1 monclient: found mon.noname-c 2024-03-29T11:19:33.680 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.695+0000 7f4699d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:19:33.681 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.695+0000 7f4699d9b640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.179:6789/0 2024-03-29T11:19:33.681 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.695+0000 7f4699d9b640 1 -- 172.21.15.179:0/3882078564 --> v1:172.21.15.179:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f469c144070 con 0x7f469c0b1780 2024-03-29T11:19:33.681 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.695+0000 7f4699d9b640 10 monclient: _finish_auth 0 2024-03-29T11:19:33.681 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.695+0000 7f4699d9b640 10 monclient: _check_auth_tickets 2024-03-29T11:19:33.681 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.695+0000 7f4699d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:19:33.681 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.695+0000 7f46a10e0640 5 monclient: authenticate success, global_id 4129 2024-03-29T11:19:33.681 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.695+0000 7f46a10e0640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:19:33.681 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.695+0000 7f4699d9b640 1 -- 172.21.15.179:0/3882078564 <== mon.1 v1:172.21.15.179:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4236868694 0 0) 0x7f4690001690 con 0x7f469c0b1780 2024-03-29T11:19:33.681 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.695+0000 7f4699d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:33.681 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.695+0000 7f4699d9b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:19:33.681 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.695+0000 7f4699d9b640 10 monclient: dump: 2024-03-29T11:19:33.681 INFO:teuthology.orchestra.run.smithi179.stderr:epoch 1 2024-03-29T11:19:33.681 INFO:teuthology.orchestra.run.smithi179.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:19:33.681 INFO:teuthology.orchestra.run.smithi179.stderr:last_changed 2024-03-29T11:18:53.864025+0000 2024-03-29T11:19:33.681 INFO:teuthology.orchestra.run.smithi179.stderr:created 2024-03-29T11:18:53.864025+0000 2024-03-29T11:19:33.682 INFO:teuthology.orchestra.run.smithi179.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:33.682 INFO:teuthology.orchestra.run.smithi179.stderr:election_strategy: 1 2024-03-29T11:19:33.682 INFO:teuthology.orchestra.run.smithi179.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T11:19:33.682 INFO:teuthology.orchestra.run.smithi179.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T11:19:33.682 INFO:teuthology.orchestra.run.smithi179.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T11:19:33.682 INFO:teuthology.orchestra.run.smithi179.stderr: 2024-03-29T11:19:33.682 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.695+0000 7f46a10e0640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:19:33.682 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.695+0000 7f4699d9b640 1 -- 172.21.15.179:0/3882078564 <== mon.1 v1:172.21.15.179:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4690001870 con 0x7f469c0b1780 2024-03-29T11:19:33.682 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.695+0000 7f4699d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:19:33.682 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.695+0000 7f4699d9b640 1 -- 172.21.15.179:0/3882078564 <== mon.1 v1:172.21.15.179:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4236868694 0 0) 0x7f4690001c80 con 0x7f469c0b1780 2024-03-29T11:19:33.682 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.695+0000 7f46a10e0640 10 monclient: get_monmap_and_config success 2024-03-29T11:19:33.682 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.695+0000 7f46a10e0640 4 set_mon_vals no callback set 2024-03-29T11:19:33.682 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.695+0000 7f4699d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:33.682 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.695+0000 7f4699d9b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:19:33.682 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.695+0000 7f4699d9b640 10 monclient: dump: 2024-03-29T11:19:33.683 INFO:teuthology.orchestra.run.smithi179.stderr:epoch 1 2024-03-29T11:19:33.683 INFO:teuthology.orchestra.run.smithi179.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:19:33.683 INFO:teuthology.orchestra.run.smithi179.stderr:last_changed 2024-03-29T11:18:53.864025+0000 2024-03-29T11:19:33.683 INFO:teuthology.orchestra.run.smithi179.stderr:created 2024-03-29T11:18:53.864025+0000 2024-03-29T11:19:33.683 INFO:teuthology.orchestra.run.smithi179.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:33.683 INFO:teuthology.orchestra.run.smithi179.stderr:election_strategy: 1 2024-03-29T11:19:33.683 INFO:teuthology.orchestra.run.smithi179.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T11:19:33.683 INFO:teuthology.orchestra.run.smithi179.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T11:19:33.683 INFO:teuthology.orchestra.run.smithi179.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T11:19:33.683 INFO:teuthology.orchestra.run.smithi179.stderr: 2024-03-29T11:19:33.683 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.695+0000 7f46a10e0640 10 monclient: shutdown 2024-03-29T11:19:33.683 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.695+0000 7f46a10e0640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:19:33.683 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.695+0000 7f46a10e0640 1 -- 172.21.15.179:0/3882078564 >> v1:172.21.15.179:6789/0 conn(0x7f469c0b1780 legacy=0x7f469c13d0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:33.683 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.695+0000 7f46a10e0640 1 -- 172.21.15.179:0/3882078564 shutdown_connections 2024-03-29T11:19:33.683 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.695+0000 7f46a10e0640 1 -- 172.21.15.179:0/3882078564 >> 172.21.15.179:0/3882078564 conn(0x7f469c0b58f0 msgr2=0x7f469c0b5d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:19:33.684 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.695+0000 7f46a10e0640 1 -- 172.21.15.179:0/3882078564 shutdown_connections 2024-03-29T11:19:33.684 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.695+0000 7f46a10e0640 1 -- 172.21.15.179:0/3882078564 wait complete. 2024-03-29T11:19:33.684 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.696+0000 7f46a10e0640 10 monclient: build_initial_monmap 2024-03-29T11:19:33.684 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.696+0000 7f46a10e0640 10 monclient: monmap: 2024-03-29T11:19:33.684 INFO:teuthology.orchestra.run.smithi179.stderr:epoch 0 2024-03-29T11:19:33.684 INFO:teuthology.orchestra.run.smithi179.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:19:33.684 INFO:teuthology.orchestra.run.smithi179.stderr:last_changed 0.000000 2024-03-29T11:19:33.684 INFO:teuthology.orchestra.run.smithi179.stderr:created 0.000000 2024-03-29T11:19:33.684 INFO:teuthology.orchestra.run.smithi179.stderr:min_mon_release 0 (unknown) 2024-03-29T11:19:33.684 INFO:teuthology.orchestra.run.smithi179.stderr:election_strategy: 1 2024-03-29T11:19:33.684 INFO:teuthology.orchestra.run.smithi179.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-29T11:19:33.684 INFO:teuthology.orchestra.run.smithi179.stderr:1: v1:172.21.15.179:6789/0 mon.noname-b 2024-03-29T11:19:33.684 INFO:teuthology.orchestra.run.smithi179.stderr:2: v1:172.21.15.84:6790/0 mon.noname-c 2024-03-29T11:19:33.684 INFO:teuthology.orchestra.run.smithi179.stderr: 2024-03-29T11:19:33.684 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.697+0000 7f46a10e0640 1 librados: starting msgr at 2024-03-29T11:19:33.685 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.697+0000 7f46a10e0640 1 librados: starting objecter 2024-03-29T11:19:33.685 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.697+0000 7f46a10e0640 1 Processor -- start 2024-03-29T11:19:33.685 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.697+0000 7f46a10e0640 1 -- start start 2024-03-29T11:19:33.685 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.697+0000 7f46a10e0640 1 librados: setting wanted keys 2024-03-29T11:19:33.685 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.697+0000 7f46a10e0640 1 librados: calling monclient init 2024-03-29T11:19:33.685 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.697+0000 7f46a10e0640 10 monclient: init 2024-03-29T11:19:33.685 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.698+0000 7f46a10e0640 10 monclient: _reopen_session rank -1 2024-03-29T11:19:33.685 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.698+0000 7f46a10e0640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:19:33.685 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.698+0000 7f46a10e0640 10 monclient(hunting): picked mon.noname-b con 0x7f469c0b1780 addr v1:172.21.15.179:6789/0 2024-03-29T11:19:33.685 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.698+0000 7f46a10e0640 10 monclient(hunting): picked mon.noname-a con 0x7f469c13d7e0 addr v1:172.21.15.84:6789/0 2024-03-29T11:19:33.685 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.698+0000 7f46a10e0640 10 monclient(hunting): picked mon.noname-c con 0x7f469c143be0 addr v1:172.21.15.84:6790/0 2024-03-29T11:19:33.685 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.698+0000 7f46a10e0640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f469c140520 con 0x7f469c13d7e0 2024-03-29T11:19:33.685 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.698+0000 7f46a10e0640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f469c13f3c0 con 0x7f469c0b1780 2024-03-29T11:19:33.685 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.698+0000 7f46a10e0640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f469c140c20 con 0x7f469c143be0 2024-03-29T11:19:33.685 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.698+0000 7f46a10e0640 10 monclient(hunting): _renew_subs 2024-03-29T11:19:33.685 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.698+0000 7f46a10e0640 10 monclient(hunting): authenticate will time out at 812.030667s 2024-03-29T11:19:33.686 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.698+0000 7f469ad9d640 1 --1- >> v1:172.21.15.179:6789/0 conn(0x7f469c0b1780 0x7f469c13eaf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.179:6789/0 says I am v1:172.21.15.179:53464/0 (socket says 172.21.15.179:53464) 2024-03-29T11:19:33.686 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.698+0000 7f469ad9d640 1 -- 172.21.15.179:0/247455894 learned_addr learned my addr 172.21.15.179:0/247455894 (peer_addr_for_me v1:172.21.15.179:0/0) 2024-03-29T11:19:33.686 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.698+0000 7f46837fe640 10 client.?.objecter ms_handle_connect 0x7f469c0b1780 2024-03-29T11:19:33.686 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.698+0000 7f46837fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:19:33.686 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.698+0000 7f46837fe640 10 client.?.objecter ms_handle_connect 0x7f469c13d7e0 2024-03-29T11:19:33.686 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.698+0000 7f46837fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:19:33.686 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.698+0000 7f46837fe640 1 -- 172.21.15.179:0/247455894 <== mon.1 v1:172.21.15.179:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3444670400 0 0) 0x7f469c13f3c0 con 0x7f469c0b1780 2024-03-29T11:19:33.686 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.698+0000 7f46837fe640 10 monclient(hunting): _negotiate 2024-03-29T11:19:33.686 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.698+0000 7f46837fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:33.686 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.698+0000 7f46837fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:33.686 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.698+0000 7f46837fe640 10 monclient(hunting): my global_id is 4132 2024-03-29T11:19:33.687 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.698+0000 7f46837fe640 1 -- 172.21.15.179:0/247455894 --> v1:172.21.15.179:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4688004eb0 con 0x7f469c0b1780 2024-03-29T11:19:33.687 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.698+0000 7f46837fe640 10 client.?.objecter ms_handle_connect 0x7f469c143be0 2024-03-29T11:19:33.688 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.698+0000 7f46837fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:19:33.688 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.698+0000 7f46837fe640 1 -- 172.21.15.179:0/247455894 <== mon.0 v1:172.21.15.84:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 906784930 0 0) 0x7f469c140520 con 0x7f469c13d7e0 2024-03-29T11:19:33.688 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.698+0000 7f46837fe640 10 monclient(hunting): _negotiate 2024-03-29T11:19:33.688 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.698+0000 7f46837fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:33.688 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.698+0000 7f46837fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:33.688 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.698+0000 7f46837fe640 10 monclient(hunting): my global_id is 4149 2024-03-29T11:19:33.688 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.698+0000 7f46837fe640 1 -- 172.21.15.179:0/247455894 --> v1:172.21.15.84:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f469c13f3c0 con 0x7f469c13d7e0 2024-03-29T11:19:33.688 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.698+0000 7f46837fe640 1 -- 172.21.15.179:0/247455894 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3670706820 0 0) 0x7f469c140c20 con 0x7f469c143be0 2024-03-29T11:19:33.688 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.699+0000 7f46837fe640 10 monclient(hunting): _negotiate 2024-03-29T11:19:33.688 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.699+0000 7f46837fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:33.688 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.699+0000 7f46837fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:33.688 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.699+0000 7f46837fe640 10 monclient(hunting): my global_id is 4148 2024-03-29T11:19:33.688 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.699+0000 7f46837fe640 1 -- 172.21.15.179:0/247455894 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f469c140520 con 0x7f469c143be0 2024-03-29T11:19:33.688 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.699+0000 7f46837fe640 1 -- 172.21.15.179:0/247455894 <== mon.1 v1:172.21.15.179:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1549346985 0 0) 0x7f4688004eb0 con 0x7f469c0b1780 2024-03-29T11:19:33.688 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.699+0000 7f46837fe640 1 -- 172.21.15.179:0/247455894 --> v1:172.21.15.179:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f469c140c20 con 0x7f469c0b1780 2024-03-29T11:19:33.689 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.699+0000 7f46837fe640 1 -- 172.21.15.179:0/247455894 <== mon.1 v1:172.21.15.179:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4236868694 0 0) 0x7f4690001bc0 con 0x7f469c0b1780 2024-03-29T11:19:33.689 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.699+0000 7f46837fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:33.689 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.699+0000 7f46837fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:19:33.689 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.699+0000 7f46837fe640 10 monclient(hunting): dump: 2024-03-29T11:19:33.689 INFO:teuthology.orchestra.run.smithi179.stderr:epoch 1 2024-03-29T11:19:33.689 INFO:teuthology.orchestra.run.smithi179.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:19:33.689 INFO:teuthology.orchestra.run.smithi179.stderr:last_changed 2024-03-29T11:18:53.864025+0000 2024-03-29T11:19:33.689 INFO:teuthology.orchestra.run.smithi179.stderr:created 2024-03-29T11:18:53.864025+0000 2024-03-29T11:19:33.689 INFO:teuthology.orchestra.run.smithi179.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:33.689 INFO:teuthology.orchestra.run.smithi179.stderr:election_strategy: 1 2024-03-29T11:19:33.689 INFO:teuthology.orchestra.run.smithi179.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T11:19:33.689 INFO:teuthology.orchestra.run.smithi179.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T11:19:33.689 INFO:teuthology.orchestra.run.smithi179.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T11:19:33.689 INFO:teuthology.orchestra.run.smithi179.stderr: 2024-03-29T11:19:33.690 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.699+0000 7f46837fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:19:33.690 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.699+0000 7f46837fe640 1 -- 172.21.15.179:0/247455894 <== mon.0 v1:172.21.15.84:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2868535786 0 0) 0x7f469c13f3c0 con 0x7f469c13d7e0 2024-03-29T11:19:33.690 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.699+0000 7f46837fe640 1 -- 172.21.15.179:0/247455894 --> v1:172.21.15.84:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4688004eb0 con 0x7f469c13d7e0 2024-03-29T11:19:33.690 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.699+0000 7f46837fe640 1 -- 172.21.15.179:0/247455894 <== mon.0 v1:172.21.15.84:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4236868694 0 0) 0x7f468c001870 con 0x7f469c13d7e0 2024-03-29T11:19:33.690 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.699+0000 7f46837fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:33.690 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.699+0000 7f46837fe640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:19:33.690 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.699+0000 7f46837fe640 10 monclient(hunting): dump: 2024-03-29T11:19:33.690 INFO:teuthology.orchestra.run.smithi179.stderr:epoch 1 2024-03-29T11:19:33.690 INFO:teuthology.orchestra.run.smithi179.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:19:33.690 INFO:teuthology.orchestra.run.smithi179.stderr:last_changed 2024-03-29T11:18:53.864025+0000 2024-03-29T11:19:33.690 INFO:teuthology.orchestra.run.smithi179.stderr:created 2024-03-29T11:18:53.864025+0000 2024-03-29T11:19:33.690 INFO:teuthology.orchestra.run.smithi179.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:33.690 INFO:teuthology.orchestra.run.smithi179.stderr:election_strategy: 1 2024-03-29T11:19:33.690 INFO:teuthology.orchestra.run.smithi179.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T11:19:33.690 INFO:teuthology.orchestra.run.smithi179.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T11:19:33.690 INFO:teuthology.orchestra.run.smithi179.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T11:19:33.691 INFO:teuthology.orchestra.run.smithi179.stderr: 2024-03-29T11:19:33.691 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.699+0000 7f46837fe640 1 -- 172.21.15.179:0/247455894 <== mon.2 v1:172.21.15.84:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2212004036 0 0) 0x7f469c140520 con 0x7f469c143be0 2024-03-29T11:19:33.691 INFO:tasks.ceph.mgr.x.smithi179.stderr:2024-03-29T11:19:33.699+0000 7f8706237040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-29T11:19:33.694 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.699+0000 7f46837fe640 1 -- 172.21.15.179:0/247455894 --> v1:172.21.15.84:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f469c13f3c0 con 0x7f469c143be0 2024-03-29T11:19:33.694 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.699+0000 7f46837fe640 1 -- 172.21.15.179:0/247455894 <== mon.2 v1:172.21.15.84:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4236868694 0 0) 0x7f4694001a70 con 0x7f469c143be0 2024-03-29T11:19:33.694 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.699+0000 7f46837fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:33.694 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.699+0000 7f46837fe640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:19:33.694 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.699+0000 7f46837fe640 10 monclient(hunting): dump: 2024-03-29T11:19:33.694 INFO:teuthology.orchestra.run.smithi179.stderr:epoch 1 2024-03-29T11:19:33.694 INFO:teuthology.orchestra.run.smithi179.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:19:33.694 INFO:teuthology.orchestra.run.smithi179.stderr:last_changed 2024-03-29T11:18:53.864025+0000 2024-03-29T11:19:33.694 INFO:teuthology.orchestra.run.smithi179.stderr:created 2024-03-29T11:18:53.864025+0000 2024-03-29T11:19:33.694 INFO:teuthology.orchestra.run.smithi179.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:33.694 INFO:teuthology.orchestra.run.smithi179.stderr:election_strategy: 1 2024-03-29T11:19:33.694 INFO:teuthology.orchestra.run.smithi179.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T11:19:33.694 INFO:teuthology.orchestra.run.smithi179.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T11:19:33.694 INFO:teuthology.orchestra.run.smithi179.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T11:19:33.695 INFO:teuthology.orchestra.run.smithi179.stderr: 2024-03-29T11:19:33.695 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.699+0000 7f46837fe640 1 -- 172.21.15.179:0/247455894 <== mon.1 v1:172.21.15.179:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3423028164 0 0) 0x7f469c140c20 con 0x7f469c0b1780 2024-03-29T11:19:33.695 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.699+0000 7f46837fe640 1 -- 172.21.15.179:0/247455894 >> v1:172.21.15.84:6790/0 conn(0x7f469c143be0 legacy=0x7f469c1d5480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:33.695 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.699+0000 7f46837fe640 1 -- 172.21.15.179:0/247455894 >> v1:172.21.15.84:6789/0 conn(0x7f469c13d7e0 legacy=0x7f469c1d4d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:33.695 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.699+0000 7f46837fe640 10 monclient: _finish_hunting 0 2024-03-29T11:19:33.695 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.699+0000 7f46837fe640 1 monclient: found mon.b 2024-03-29T11:19:33.695 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.699+0000 7f46837fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:19:33.695 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.699+0000 7f46837fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-29T11:19:33.695 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.699+0000 7f46837fe640 1 -- 172.21.15.179:0/247455894 --> v1:172.21.15.179:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f469c140db0 con 0x7f469c0b1780 2024-03-29T11:19:33.695 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.699+0000 7f46837fe640 10 monclient: _finish_auth 0 2024-03-29T11:19:33.695 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.699+0000 7f46837fe640 10 monclient: _check_auth_tickets 2024-03-29T11:19:33.695 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.699+0000 7f46837fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:19:33.695 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.699+0000 7f46a10e0640 5 monclient: authenticate success, global_id 4132 2024-03-29T11:19:33.695 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.699+0000 7f46a10e0640 10 monclient: _renew_subs 2024-03-29T11:19:33.695 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.699+0000 7f46a10e0640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-29T11:19:33.696 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.699+0000 7f46a10e0640 1 -- 172.21.15.179:0/247455894 --> v1:172.21.15.179:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f469c0aed20 con 0x7f469c0b1780 2024-03-29T11:19:33.696 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.699+0000 7f46a10e0640 10 client.4132.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:19:33.696 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.699+0000 7f46a10e0640 10 monclient: _renew_subs 2024-03-29T11:19:33.696 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.699+0000 7f46a10e0640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-29T11:19:33.696 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.699+0000 7f46a10e0640 1 -- 172.21.15.179:0/247455894 --> v1:172.21.15.179:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f469c0af2a0 con 0x7f469c0b1780 2024-03-29T11:19:33.696 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.699+0000 7f46a10e0640 1 librados: init done 2024-03-29T11:19:33.696 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.699+0000 7f46837fe640 1 -- 172.21.15.179:0/247455894 <== mon.1 v1:172.21.15.179:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4690004bf0 con 0x7f469c0b1780 2024-03-29T11:19:33.696 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.699+0000 7f46837fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:19:33.696 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.699+0000 7f46837fe640 1 -- 172.21.15.179:0/247455894 <== mon.1 v1:172.21.15.179:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4236868694 0 0) 0x7f4690004f00 con 0x7f469c0b1780 2024-03-29T11:19:33.696 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.699+0000 7f46837fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:33.696 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.699+0000 7f46837fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:19:33.696 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.699+0000 7f46837fe640 10 monclient: dump: 2024-03-29T11:19:33.696 INFO:teuthology.orchestra.run.smithi179.stderr:epoch 1 2024-03-29T11:19:33.696 INFO:teuthology.orchestra.run.smithi179.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:19:33.696 INFO:teuthology.orchestra.run.smithi179.stderr:last_changed 2024-03-29T11:18:53.864025+0000 2024-03-29T11:19:33.697 INFO:teuthology.orchestra.run.smithi179.stderr:created 2024-03-29T11:18:53.864025+0000 2024-03-29T11:19:33.697 INFO:teuthology.orchestra.run.smithi179.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:33.697 INFO:teuthology.orchestra.run.smithi179.stderr:election_strategy: 1 2024-03-29T11:19:33.697 INFO:teuthology.orchestra.run.smithi179.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T11:19:33.697 INFO:teuthology.orchestra.run.smithi179.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T11:19:33.697 INFO:teuthology.orchestra.run.smithi179.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T11:19:33.697 INFO:teuthology.orchestra.run.smithi179.stderr: 2024-03-29T11:19:33.697 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.699+0000 7f469959a640 4 set_mon_vals no callback set 2024-03-29T11:19:33.697 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.699+0000 7f46837fe640 1 -- 172.21.15.179:0/247455894 <== mon.1 v1:172.21.15.179:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f46900053f0 con 0x7f469c0b1780 2024-03-29T11:19:33.697 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.699+0000 7f46837fe640 1 -- 172.21.15.179:0/247455894 <== mon.1 v1:172.21.15.179:6789/0 8 ==== osd_map(7..7 src has 1..7) v4 ==== 1300+0+0 (unknown 2368534226 0 0) 0x7f4690005c20 con 0x7f469c0b1780 2024-03-29T11:19:33.697 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.699+0000 7f46837fe640 10 client.4132.objecter ms_dispatch 0x7f469c000b90 osd_map(7..7 src has 1..7) v4 2024-03-29T11:19:33.697 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.699+0000 7f46837fe640 3 client.4132.objecter handle_osd_map got epochs [7,7] > 0 2024-03-29T11:19:33.697 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.699+0000 7f46837fe640 3 client.4132.objecter handle_osd_map decoding full epoch 7 2024-03-29T11:19:33.698 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.699+0000 7f46837fe640 20 client.4132.objecter dump_active .. 0 homeless 2024-03-29T11:19:33.698 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.700+0000 7f46a10e0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:33.698 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.700+0000 7f46a10e0640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:33.698 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.700+0000 7f46a10e0640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-29T11:19:33.698 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.700+0000 7f46a10e0640 1 -- 172.21.15.179:0/247455894 --> v1:172.21.15.179:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f467c005080 con 0x7f469c0b1780 2024-03-29T11:19:33.698 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.702+0000 7f46837fe640 1 -- 172.21.15.179:0/247455894 <== mon.1 v1:172.21.15.179:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f4690005de0 con 0x7f469c0b1780 2024-03-29T11:19:33.698 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.702+0000 7f46837fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:33.698 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.702+0000 7f46837fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:19:33.724 INFO:tasks.ceph.mgr.x.smithi179.stderr:2024-03-29T11:19:33.748+0000 7f8706237040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-29T11:19:33.751 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.775+0000 7f46a10e0640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "f0cf44ea-6055-48af-805b-aed858102ff9", "id": 5}] 2024-03-29T11:19:33.752 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.775+0000 7f46a10e0640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "f0cf44ea-6055-48af-805b-aed858102ff9", "id": 5}] 2024-03-29T11:19:33.752 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.775+0000 7f46a10e0640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-29T11:19:33.752 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.775+0000 7f46a10e0640 1 -- 172.21.15.179:0/247455894 --> v1:172.21.15.179:6789/0 -- mon_command({"prefix": "osd new", "uuid": "f0cf44ea-6055-48af-805b-aed858102ff9", "id": 5} v 0) v1 -- 0x7f467c0058b0 con 0x7f469c0b1780 2024-03-29T11:19:33.755 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.779+0000 7f46837fe640 1 -- 172.21.15.179:0/247455894 <== mon.1 v1:172.21.15.179:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "f0cf44ea-6055-48af-805b-aed858102ff9", "id": 5}]=0 v8) v1 ==== 112+0+2 (unknown 2780877425 0 1988412176) 0x7f46900056a0 con 0x7f469c0b1780 2024-03-29T11:19:33.755 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.779+0000 7f46837fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "f0cf44ea-6055-48af-805b-aed858102ff9", "id": 5}] 2024-03-29T11:19:33.755 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.779+0000 7f46837fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:19:33.756 INFO:teuthology.orchestra.run.smithi179.stdout:5 2024-03-29T11:19:33.757 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.780+0000 7f46a10e0640 10 librados: watch_flush enter 2024-03-29T11:19:33.757 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.780+0000 7f46a10e0640 10 librados: watch_flush exit 2024-03-29T11:19:33.757 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.780+0000 7f46a10e0640 20 client.4132.objecter shutdown clearing up homeless session... 2024-03-29T11:19:33.757 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.780+0000 7f46a10e0640 10 client.4132.objecter successfully canceled tick 2024-03-29T11:19:33.757 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.780+0000 7f46a10e0640 10 monclient: shutdown 2024-03-29T11:19:33.757 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.780+0000 7f46a10e0640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:19:33.757 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.780+0000 7f46a10e0640 1 -- 172.21.15.179:0/247455894 >> v1:172.21.15.179:6789/0 conn(0x7f469c0b1780 legacy=0x7f469c13eaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:33.757 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.780+0000 7f46a10e0640 1 -- 172.21.15.179:0/247455894 shutdown_connections 2024-03-29T11:19:33.757 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.780+0000 7f46a10e0640 1 -- 172.21.15.179:0/247455894 >> 172.21.15.179:0/247455894 conn(0x7f469c0b58f0 msgr2=0x7f469c069bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:19:33.757 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.780+0000 7f46a10e0640 1 -- 172.21.15.179:0/247455894 shutdown_connections 2024-03-29T11:19:33.757 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.780+0000 7f46a10e0640 1 -- 172.21.15.179:0/247455894 wait complete. 2024-03-29T11:19:33.758 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.780+0000 7f46a10e0640 1 librados: shutdown 2024-03-29T11:19:33.767 DEBUG:teuthology.orchestra.run.smithi179:> sudo ceph --cluster ceph osd new d895eaa0-cc0b-4ee4-9b5a-2dfcd89a60b7 6 2024-03-29T11:19:33.867 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.890+0000 7fc6c36d9640 10 monclient: get_monmap_and_config 2024-03-29T11:19:33.867 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.890+0000 7fc6c36d9640 10 monclient: build_initial_monmap 2024-03-29T11:19:33.867 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.890+0000 7fc6c36d9640 10 monclient: monmap: 2024-03-29T11:19:33.867 INFO:teuthology.orchestra.run.smithi179.stderr:epoch 0 2024-03-29T11:19:33.867 INFO:teuthology.orchestra.run.smithi179.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:19:33.867 INFO:teuthology.orchestra.run.smithi179.stderr:last_changed 2024-03-29T11:19:33.891035+0000 2024-03-29T11:19:33.867 INFO:teuthology.orchestra.run.smithi179.stderr:created 2024-03-29T11:19:33.891035+0000 2024-03-29T11:19:33.867 INFO:teuthology.orchestra.run.smithi179.stderr:min_mon_release 0 (unknown) 2024-03-29T11:19:33.868 INFO:teuthology.orchestra.run.smithi179.stderr:election_strategy: 1 2024-03-29T11:19:33.868 INFO:teuthology.orchestra.run.smithi179.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-29T11:19:33.868 INFO:teuthology.orchestra.run.smithi179.stderr:1: v1:172.21.15.179:6789/0 mon.noname-c 2024-03-29T11:19:33.868 INFO:teuthology.orchestra.run.smithi179.stderr:2: v1:172.21.15.84:6790/0 mon.noname-b 2024-03-29T11:19:33.868 INFO:teuthology.orchestra.run.smithi179.stderr: 2024-03-29T11:19:33.868 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.890+0000 7fc6c36d9640 1 Processor -- start 2024-03-29T11:19:33.868 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.890+0000 7fc6c36d9640 1 -- start start 2024-03-29T11:19:33.868 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.890+0000 7fc6c36d9640 10 monclient: init 2024-03-29T11:19:33.877 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.891+0000 7fc6c36d9640 10 monclient: _reopen_session rank -1 2024-03-29T11:19:33.877 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.891+0000 7fc6c36d9640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:19:33.877 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.891+0000 7fc6c36d9640 10 monclient(hunting): picked mon.noname-b con 0x7fc6bc09ac40 addr v1:172.21.15.84:6790/0 2024-03-29T11:19:33.877 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.891+0000 7fc6c36d9640 10 monclient(hunting): picked mon.noname-c con 0x7fc6bc09c870 addr v1:172.21.15.179:6789/0 2024-03-29T11:19:33.877 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.891+0000 7fc6c36d9640 10 monclient(hunting): picked mon.noname-a con 0x7fc6bc0b4360 addr v1:172.21.15.84:6789/0 2024-03-29T11:19:33.877 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.891+0000 7fc6c36d9640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc6bc09dec0 con 0x7fc6bc0b4360 2024-03-29T11:19:33.877 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.891+0000 7fc6c36d9640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc6bc09da30 con 0x7fc6bc09c870 2024-03-29T11:19:33.877 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.891+0000 7fc6c36d9640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc6bc0b39c0 con 0x7fc6bc09ac40 2024-03-29T11:19:33.877 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.891+0000 7fc6c36d9640 10 monclient(hunting): _renew_subs 2024-03-29T11:19:33.877 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.891+0000 7fc6c36d9640 10 monclient(hunting): authenticate will time out at 812.223661s 2024-03-29T11:19:33.877 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.891+0000 7fc6c0c74640 1 --1- >> v1:172.21.15.179:6789/0 conn(0x7fc6bc09c870 0x7fc6bc13f3a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.179:6789/0 says I am v1:172.21.15.179:53476/0 (socket says 172.21.15.179:53476) 2024-03-29T11:19:33.877 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.891+0000 7fc6c0c74640 1 -- 172.21.15.179:0/1189743978 learned_addr learned my addr 172.21.15.179:0/1189743978 (peer_addr_for_me v1:172.21.15.179:0/0) 2024-03-29T11:19:33.878 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.891+0000 7fc6b3fff640 1 -- 172.21.15.179:0/1189743978 <== mon.1 v1:172.21.15.179:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1024392084 0 0) 0x7fc6bc09da30 con 0x7fc6bc09c870 2024-03-29T11:19:33.878 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.891+0000 7fc6b3fff640 10 monclient(hunting): _negotiate 2024-03-29T11:19:33.878 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.891+0000 7fc6b3fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:33.878 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.891+0000 7fc6b3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:33.878 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.891+0000 7fc6b3fff640 10 monclient(hunting): my global_id is 4135 2024-03-29T11:19:33.878 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.891+0000 7fc6b3fff640 1 -- 172.21.15.179:0/1189743978 --> v1:172.21.15.179:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc6a8003520 con 0x7fc6bc09c870 2024-03-29T11:19:33.878 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.892+0000 7fc6b3fff640 1 -- 172.21.15.179:0/1189743978 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3462047837 0 0) 0x7fc6bc0b39c0 con 0x7fc6bc09ac40 2024-03-29T11:19:33.878 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.892+0000 7fc6b3fff640 10 monclient(hunting): _negotiate 2024-03-29T11:19:33.878 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.892+0000 7fc6b3fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:33.878 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.892+0000 7fc6b3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:33.878 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.892+0000 7fc6b3fff640 10 monclient(hunting): my global_id is 4151 2024-03-29T11:19:33.878 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.892+0000 7fc6b3fff640 1 -- 172.21.15.179:0/1189743978 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc6bc09da30 con 0x7fc6bc09ac40 2024-03-29T11:19:33.878 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.892+0000 7fc6b3fff640 1 -- 172.21.15.179:0/1189743978 <== mon.0 v1:172.21.15.84:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 314917876 0 0) 0x7fc6b4001490 con 0x7fc6bc0b4360 2024-03-29T11:19:33.878 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.892+0000 7fc6b3fff640 10 monclient(hunting): _negotiate 2024-03-29T11:19:33.879 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.892+0000 7fc6b3fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:33.879 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.892+0000 7fc6b3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:33.879 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.892+0000 7fc6b3fff640 10 monclient(hunting): my global_id is 4152 2024-03-29T11:19:33.879 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.892+0000 7fc6b3fff640 1 -- 172.21.15.179:0/1189743978 --> v1:172.21.15.84:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc6bc0b39c0 con 0x7fc6bc0b4360 2024-03-29T11:19:33.879 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.892+0000 7fc6b3fff640 1 -- 172.21.15.179:0/1189743978 <== mon.1 v1:172.21.15.179:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 274619237 0 0) 0x7fc6a8003520 con 0x7fc6bc09c870 2024-03-29T11:19:33.879 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.892+0000 7fc6b3fff640 1 -- 172.21.15.179:0/1189743978 >> v1:172.21.15.84:6790/0 conn(0x7fc6bc09ac40 legacy=0x7fc6bc0b4a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:33.879 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.892+0000 7fc6b3fff640 1 -- 172.21.15.179:0/1189743978 >> v1:172.21.15.84:6789/0 conn(0x7fc6bc0b4360 legacy=0x7fc6bc14d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:33.879 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.892+0000 7fc6b3fff640 10 monclient: _finish_hunting 0 2024-03-29T11:19:33.879 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.892+0000 7fc6b3fff640 1 monclient: found mon.noname-c 2024-03-29T11:19:33.879 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.892+0000 7fc6b3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:19:33.879 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.892+0000 7fc6b3fff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.179:6789/0 2024-03-29T11:19:33.879 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.892+0000 7fc6b3fff640 1 -- 172.21.15.179:0/1189743978 --> v1:172.21.15.179:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6bc0b5160 con 0x7fc6bc09c870 2024-03-29T11:19:33.879 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.892+0000 7fc6b3fff640 10 monclient: _finish_auth 0 2024-03-29T11:19:33.879 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.892+0000 7fc6b3fff640 10 monclient: _check_auth_tickets 2024-03-29T11:19:33.879 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.892+0000 7fc6b3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:19:33.880 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.892+0000 7fc6b3fff640 1 -- 172.21.15.179:0/1189743978 <== mon.1 v1:172.21.15.179:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4236868694 0 0) 0x7fc6ac001690 con 0x7fc6bc09c870 2024-03-29T11:19:33.880 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.892+0000 7fc6c36d9640 5 monclient: authenticate success, global_id 4135 2024-03-29T11:19:33.880 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.892+0000 7fc6c36d9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:19:33.880 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.892+0000 7fc6b3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:33.880 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.892+0000 7fc6b3fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:19:33.880 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.892+0000 7fc6b3fff640 10 monclient: dump: 2024-03-29T11:19:33.880 INFO:teuthology.orchestra.run.smithi179.stderr:epoch 1 2024-03-29T11:19:33.880 INFO:teuthology.orchestra.run.smithi179.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:19:33.880 INFO:teuthology.orchestra.run.smithi179.stderr:last_changed 2024-03-29T11:18:53.864025+0000 2024-03-29T11:19:33.880 INFO:teuthology.orchestra.run.smithi179.stderr:created 2024-03-29T11:18:53.864025+0000 2024-03-29T11:19:33.880 INFO:teuthology.orchestra.run.smithi179.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:33.880 INFO:teuthology.orchestra.run.smithi179.stderr:election_strategy: 1 2024-03-29T11:19:33.880 INFO:teuthology.orchestra.run.smithi179.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T11:19:33.880 INFO:teuthology.orchestra.run.smithi179.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T11:19:33.881 INFO:teuthology.orchestra.run.smithi179.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T11:19:33.881 INFO:teuthology.orchestra.run.smithi179.stderr: 2024-03-29T11:19:33.881 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.892+0000 7fc6c36d9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:19:33.881 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.892+0000 7fc6b3fff640 1 -- 172.21.15.179:0/1189743978 <== mon.1 v1:172.21.15.179:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc6ac0017f0 con 0x7fc6bc09c870 2024-03-29T11:19:33.881 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.892+0000 7fc6b3fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:19:33.881 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.892+0000 7fc6c36d9640 10 monclient: get_monmap_and_config success 2024-03-29T11:19:33.881 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.892+0000 7fc6b3fff640 1 -- 172.21.15.179:0/1189743978 <== mon.1 v1:172.21.15.179:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4236868694 0 0) 0x7fc6ac001c00 con 0x7fc6bc09c870 2024-03-29T11:19:33.881 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.892+0000 7fc6b3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:33.881 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.892+0000 7fc6c36d9640 4 set_mon_vals no callback set 2024-03-29T11:19:33.881 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.892+0000 7fc6b3fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:19:33.881 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.892+0000 7fc6b3fff640 10 monclient: dump: 2024-03-29T11:19:33.881 INFO:teuthology.orchestra.run.smithi179.stderr:epoch 1 2024-03-29T11:19:33.881 INFO:teuthology.orchestra.run.smithi179.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:19:33.881 INFO:teuthology.orchestra.run.smithi179.stderr:last_changed 2024-03-29T11:18:53.864025+0000 2024-03-29T11:19:33.881 INFO:teuthology.orchestra.run.smithi179.stderr:created 2024-03-29T11:18:53.864025+0000 2024-03-29T11:19:33.882 INFO:teuthology.orchestra.run.smithi179.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:33.882 INFO:teuthology.orchestra.run.smithi179.stderr:election_strategy: 1 2024-03-29T11:19:33.882 INFO:teuthology.orchestra.run.smithi179.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T11:19:33.882 INFO:teuthology.orchestra.run.smithi179.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T11:19:33.882 INFO:teuthology.orchestra.run.smithi179.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T11:19:33.882 INFO:teuthology.orchestra.run.smithi179.stderr: 2024-03-29T11:19:33.882 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.892+0000 7fc6c36d9640 10 monclient: shutdown 2024-03-29T11:19:33.882 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.892+0000 7fc6c36d9640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:19:33.882 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.892+0000 7fc6c36d9640 1 -- 172.21.15.179:0/1189743978 >> v1:172.21.15.179:6789/0 conn(0x7fc6bc09c870 legacy=0x7fc6bc13f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:33.882 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.892+0000 7fc6c36d9640 1 -- 172.21.15.179:0/1189743978 shutdown_connections 2024-03-29T11:19:33.882 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.892+0000 7fc6c36d9640 1 -- 172.21.15.179:0/1189743978 >> 172.21.15.179:0/1189743978 conn(0x7fc6bc066b00 msgr2=0x7fc6bc066f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:19:33.882 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.893+0000 7fc6c36d9640 1 -- 172.21.15.179:0/1189743978 shutdown_connections 2024-03-29T11:19:33.882 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.893+0000 7fc6c36d9640 1 -- 172.21.15.179:0/1189743978 wait complete. 2024-03-29T11:19:33.882 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.893+0000 7fc6c36d9640 10 monclient: build_initial_monmap 2024-03-29T11:19:33.882 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.893+0000 7fc6c36d9640 10 monclient: monmap: 2024-03-29T11:19:33.883 INFO:teuthology.orchestra.run.smithi179.stderr:epoch 0 2024-03-29T11:19:33.883 INFO:teuthology.orchestra.run.smithi179.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:19:33.883 INFO:teuthology.orchestra.run.smithi179.stderr:last_changed 0.000000 2024-03-29T11:19:33.883 INFO:teuthology.orchestra.run.smithi179.stderr:created 0.000000 2024-03-29T11:19:33.883 INFO:teuthology.orchestra.run.smithi179.stderr:min_mon_release 0 (unknown) 2024-03-29T11:19:33.883 INFO:teuthology.orchestra.run.smithi179.stderr:election_strategy: 1 2024-03-29T11:19:33.883 INFO:teuthology.orchestra.run.smithi179.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-29T11:19:33.883 INFO:teuthology.orchestra.run.smithi179.stderr:1: v1:172.21.15.179:6789/0 mon.noname-b 2024-03-29T11:19:33.883 INFO:teuthology.orchestra.run.smithi179.stderr:2: v1:172.21.15.84:6790/0 mon.noname-c 2024-03-29T11:19:33.883 INFO:teuthology.orchestra.run.smithi179.stderr: 2024-03-29T11:19:33.883 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.893+0000 7fc6c36d9640 1 librados: starting msgr at 2024-03-29T11:19:33.883 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.893+0000 7fc6c36d9640 1 librados: starting objecter 2024-03-29T11:19:33.883 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.893+0000 7fc6c36d9640 1 Processor -- start 2024-03-29T11:19:33.883 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.894+0000 7fc6c36d9640 1 -- start start 2024-03-29T11:19:33.883 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.894+0000 7fc6c36d9640 1 librados: setting wanted keys 2024-03-29T11:19:33.884 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.894+0000 7fc6c36d9640 1 librados: calling monclient init 2024-03-29T11:19:33.884 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.894+0000 7fc6c36d9640 10 monclient: init 2024-03-29T11:19:33.884 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.894+0000 7fc6c36d9640 10 monclient: _reopen_session rank -1 2024-03-29T11:19:33.884 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.894+0000 7fc6c36d9640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:19:33.884 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.894+0000 7fc6c36d9640 10 monclient(hunting): picked mon.noname-b con 0x7fc6bc09ac40 addr v1:172.21.15.179:6789/0 2024-03-29T11:19:33.884 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.894+0000 7fc6c36d9640 10 monclient(hunting): picked mon.noname-a con 0x7fc6bc0b8130 addr v1:172.21.15.84:6789/0 2024-03-29T11:19:33.884 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.894+0000 7fc6c36d9640 10 monclient(hunting): picked mon.noname-c con 0x7fc6bc13ffa0 addr v1:172.21.15.84:6790/0 2024-03-29T11:19:33.884 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.894+0000 7fc6c36d9640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc6bc1d72b0 con 0x7fc6bc0b8130 2024-03-29T11:19:33.884 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.894+0000 7fc6c36d9640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc6bc1d7590 con 0x7fc6bc09ac40 2024-03-29T11:19:33.884 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.894+0000 7fc6c36d9640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc6bc1d8130 con 0x7fc6bc13ffa0 2024-03-29T11:19:33.884 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.894+0000 7fc6c36d9640 10 monclient(hunting): _renew_subs 2024-03-29T11:19:33.884 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.894+0000 7fc6c36d9640 10 monclient(hunting): authenticate will time out at 812.226641s 2024-03-29T11:19:33.884 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.894+0000 7fc6c1475640 1 --1- >> v1:172.21.15.179:6789/0 conn(0x7fc6bc09ac40 0x7fc6bc142a50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.179:6789/0 says I am v1:172.21.15.179:53492/0 (socket says 172.21.15.179:53492) 2024-03-29T11:19:33.884 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.894+0000 7fc6c1475640 1 -- 172.21.15.179:0/689965376 learned_addr learned my addr 172.21.15.179:0/689965376 (peer_addr_for_me v1:172.21.15.179:0/0) 2024-03-29T11:19:33.884 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.894+0000 7fc6b1ffb640 10 client.?.objecter ms_handle_connect 0x7fc6bc09ac40 2024-03-29T11:19:33.885 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.894+0000 7fc6b1ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:19:33.885 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.894+0000 7fc6b1ffb640 10 client.?.objecter ms_handle_connect 0x7fc6bc0b8130 2024-03-29T11:19:33.885 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.894+0000 7fc6b1ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:19:33.885 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.894+0000 7fc6b1ffb640 10 client.?.objecter ms_handle_connect 0x7fc6bc13ffa0 2024-03-29T11:19:33.885 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.894+0000 7fc6b1ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:19:33.885 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.894+0000 7fc6b1ffb640 1 -- 172.21.15.179:0/689965376 <== mon.1 v1:172.21.15.179:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2264279166 0 0) 0x7fc6bc1d7590 con 0x7fc6bc09ac40 2024-03-29T11:19:33.885 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.894+0000 7fc6b1ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:19:33.885 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.894+0000 7fc6b1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:33.885 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.894+0000 7fc6b1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:33.885 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.894+0000 7fc6b1ffb640 10 monclient(hunting): my global_id is 4138 2024-03-29T11:19:33.885 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.894+0000 7fc6b1ffb640 1 -- 172.21.15.179:0/689965376 --> v1:172.21.15.179:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc6a0004eb0 con 0x7fc6bc09ac40 2024-03-29T11:19:33.885 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.894+0000 7fc6b1ffb640 1 -- 172.21.15.179:0/689965376 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2442957966 0 0) 0x7fc6bc1d8130 con 0x7fc6bc13ffa0 2024-03-29T11:19:33.885 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.894+0000 7fc6b1ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:19:33.885 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.894+0000 7fc6b1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:33.886 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.894+0000 7fc6b1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:33.886 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.894+0000 7fc6b1ffb640 10 monclient(hunting): my global_id is 4154 2024-03-29T11:19:33.886 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.894+0000 7fc6b1ffb640 1 -- 172.21.15.179:0/689965376 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc6bc1d7590 con 0x7fc6bc13ffa0 2024-03-29T11:19:33.886 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.894+0000 7fc6b1ffb640 1 -- 172.21.15.179:0/689965376 <== mon.0 v1:172.21.15.84:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3280755459 0 0) 0x7fc6bc1d72b0 con 0x7fc6bc0b8130 2024-03-29T11:19:33.886 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.894+0000 7fc6b1ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:19:33.886 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.894+0000 7fc6b1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:33.886 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.894+0000 7fc6b1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:33.886 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.894+0000 7fc6b1ffb640 10 monclient(hunting): my global_id is 4155 2024-03-29T11:19:33.886 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.895+0000 7fc6b1ffb640 1 -- 172.21.15.179:0/689965376 --> v1:172.21.15.84:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc6bc1d8130 con 0x7fc6bc0b8130 2024-03-29T11:19:33.886 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.895+0000 7fc6b1ffb640 1 -- 172.21.15.179:0/689965376 <== mon.1 v1:172.21.15.179:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 915046322 0 0) 0x7fc6a0004eb0 con 0x7fc6bc09ac40 2024-03-29T11:19:33.886 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.895+0000 7fc6b1ffb640 1 -- 172.21.15.179:0/689965376 --> v1:172.21.15.179:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc6bc1d72b0 con 0x7fc6bc09ac40 2024-03-29T11:19:33.886 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.895+0000 7fc6b1ffb640 1 -- 172.21.15.179:0/689965376 <== mon.1 v1:172.21.15.179:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4236868694 0 0) 0x7fc6b8001a30 con 0x7fc6bc09ac40 2024-03-29T11:19:33.887 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.895+0000 7fc6b1ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:33.888 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.895+0000 7fc6b1ffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:19:33.888 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.895+0000 7fc6b1ffb640 10 monclient(hunting): dump: 2024-03-29T11:19:33.888 INFO:teuthology.orchestra.run.smithi179.stderr:epoch 1 2024-03-29T11:19:33.888 INFO:teuthology.orchestra.run.smithi179.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:19:33.888 INFO:teuthology.orchestra.run.smithi179.stderr:last_changed 2024-03-29T11:18:53.864025+0000 2024-03-29T11:19:33.888 INFO:teuthology.orchestra.run.smithi179.stderr:created 2024-03-29T11:18:53.864025+0000 2024-03-29T11:19:33.888 INFO:teuthology.orchestra.run.smithi179.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:33.888 INFO:teuthology.orchestra.run.smithi179.stderr:election_strategy: 1 2024-03-29T11:19:33.888 INFO:teuthology.orchestra.run.smithi179.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T11:19:33.888 INFO:teuthology.orchestra.run.smithi179.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T11:19:33.888 INFO:teuthology.orchestra.run.smithi179.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T11:19:33.888 INFO:teuthology.orchestra.run.smithi179.stderr: 2024-03-29T11:19:33.888 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.895+0000 7fc6b1ffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:19:33.888 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.895+0000 7fc6b1ffb640 1 -- 172.21.15.179:0/689965376 <== mon.2 v1:172.21.15.84:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4149047618 0 0) 0x7fc6bc1d7590 con 0x7fc6bc13ffa0 2024-03-29T11:19:33.889 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.895+0000 7fc6b1ffb640 1 -- 172.21.15.179:0/689965376 --> v1:172.21.15.84:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc6a0004eb0 con 0x7fc6bc13ffa0 2024-03-29T11:19:33.889 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.895+0000 7fc6b1ffb640 1 -- 172.21.15.179:0/689965376 <== mon.2 v1:172.21.15.84:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4236868694 0 0) 0x7fc6b4001b40 con 0x7fc6bc13ffa0 2024-03-29T11:19:33.889 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.895+0000 7fc6b1ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:33.889 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.895+0000 7fc6b1ffb640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:19:33.889 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.895+0000 7fc6b1ffb640 10 monclient(hunting): dump: 2024-03-29T11:19:33.889 INFO:teuthology.orchestra.run.smithi179.stderr:epoch 1 2024-03-29T11:19:33.889 INFO:teuthology.orchestra.run.smithi179.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:19:33.889 INFO:teuthology.orchestra.run.smithi179.stderr:last_changed 2024-03-29T11:18:53.864025+0000 2024-03-29T11:19:33.889 INFO:teuthology.orchestra.run.smithi179.stderr:created 2024-03-29T11:18:53.864025+0000 2024-03-29T11:19:33.889 INFO:teuthology.orchestra.run.smithi179.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:33.889 INFO:teuthology.orchestra.run.smithi179.stderr:election_strategy: 1 2024-03-29T11:19:33.889 INFO:teuthology.orchestra.run.smithi179.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T11:19:33.889 INFO:teuthology.orchestra.run.smithi179.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T11:19:33.889 INFO:teuthology.orchestra.run.smithi179.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T11:19:33.890 INFO:teuthology.orchestra.run.smithi179.stderr: 2024-03-29T11:19:33.890 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.895+0000 7fc6b1ffb640 1 -- 172.21.15.179:0/689965376 <== mon.0 v1:172.21.15.84:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1422424094 0 0) 0x7fc6bc1d8130 con 0x7fc6bc0b8130 2024-03-29T11:19:33.890 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.895+0000 7fc6b1ffb640 1 -- 172.21.15.179:0/689965376 --> v1:172.21.15.84:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc6bc1d7590 con 0x7fc6bc0b8130 2024-03-29T11:19:33.890 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.895+0000 7fc6b1ffb640 1 -- 172.21.15.179:0/689965376 <== mon.1 v1:172.21.15.179:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3498787839 0 0) 0x7fc6bc1d72b0 con 0x7fc6bc09ac40 2024-03-29T11:19:33.890 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.895+0000 7fc6b1ffb640 1 -- 172.21.15.179:0/689965376 >> v1:172.21.15.84:6790/0 conn(0x7fc6bc13ffa0 legacy=0x7fc6bc140750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:33.890 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.895+0000 7fc6b1ffb640 1 -- 172.21.15.179:0/689965376 >> v1:172.21.15.84:6789/0 conn(0x7fc6bc0b8130 legacy=0x7fc6bc1d7a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:33.890 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.895+0000 7fc6b1ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:19:33.890 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.895+0000 7fc6b1ffb640 1 monclient: found mon.b 2024-03-29T11:19:33.890 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.895+0000 7fc6b1ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:19:33.890 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.895+0000 7fc6b1ffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-29T11:19:33.890 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.895+0000 7fc6b1ffb640 1 -- 172.21.15.179:0/689965376 --> v1:172.21.15.179:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6bc141fe0 con 0x7fc6bc09ac40 2024-03-29T11:19:33.890 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.895+0000 7fc6b1ffb640 10 monclient: _finish_auth 0 2024-03-29T11:19:33.890 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.895+0000 7fc6b1ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:19:33.891 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.895+0000 7fc6b1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:19:33.891 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.895+0000 7fc6c36d9640 5 monclient: authenticate success, global_id 4138 2024-03-29T11:19:33.891 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.895+0000 7fc6c36d9640 10 monclient: _renew_subs 2024-03-29T11:19:33.891 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.895+0000 7fc6c36d9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-29T11:19:33.891 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.895+0000 7fc6c36d9640 1 -- 172.21.15.179:0/689965376 --> v1:172.21.15.179:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc6bc140e60 con 0x7fc6bc09ac40 2024-03-29T11:19:33.891 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.895+0000 7fc6b1ffb640 1 -- 172.21.15.179:0/689965376 <== mon.1 v1:172.21.15.179:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc6b8001cf0 con 0x7fc6bc09ac40 2024-03-29T11:19:33.891 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.895+0000 7fc6b1ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:19:33.891 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.895+0000 7fc6b1ffb640 1 -- 172.21.15.179:0/689965376 <== mon.1 v1:172.21.15.179:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4236868694 0 0) 0x7fc6b8003a60 con 0x7fc6bc09ac40 2024-03-29T11:19:33.891 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.895+0000 7fc6b1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:33.891 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.895+0000 7fc6b1ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:19:33.891 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.895+0000 7fc6b1ffb640 10 monclient: dump: 2024-03-29T11:19:33.891 INFO:teuthology.orchestra.run.smithi179.stderr:epoch 1 2024-03-29T11:19:33.891 INFO:teuthology.orchestra.run.smithi179.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:19:33.891 INFO:teuthology.orchestra.run.smithi179.stderr:last_changed 2024-03-29T11:18:53.864025+0000 2024-03-29T11:19:33.891 INFO:teuthology.orchestra.run.smithi179.stderr:created 2024-03-29T11:18:53.864025+0000 2024-03-29T11:19:33.891 INFO:teuthology.orchestra.run.smithi179.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:33.892 INFO:teuthology.orchestra.run.smithi179.stderr:election_strategy: 1 2024-03-29T11:19:33.892 INFO:teuthology.orchestra.run.smithi179.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T11:19:33.892 INFO:teuthology.orchestra.run.smithi179.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T11:19:33.892 INFO:teuthology.orchestra.run.smithi179.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T11:19:33.892 INFO:teuthology.orchestra.run.smithi179.stderr: 2024-03-29T11:19:33.892 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.895+0000 7fc6b37fe640 4 set_mon_vals no callback set 2024-03-29T11:19:33.892 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.895+0000 7fc6c36d9640 10 client.4138.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:19:33.892 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.895+0000 7fc6c36d9640 10 monclient: _renew_subs 2024-03-29T11:19:33.892 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.895+0000 7fc6c36d9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-29T11:19:33.892 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.895+0000 7fc6b1ffb640 1 -- 172.21.15.179:0/689965376 <== mon.1 v1:172.21.15.179:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7fc6b8003f50 con 0x7fc6bc09ac40 2024-03-29T11:19:33.892 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.895+0000 7fc6c36d9640 1 -- 172.21.15.179:0/689965376 --> v1:172.21.15.179:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc6bc1e3b00 con 0x7fc6bc09ac40 2024-03-29T11:19:33.892 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.895+0000 7fc6c36d9640 1 librados: init done 2024-03-29T11:19:33.892 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.895+0000 7fc6b1ffb640 1 -- 172.21.15.179:0/689965376 <== mon.1 v1:172.21.15.179:6789/0 8 ==== osd_map(8..8 src has 1..8) v4 ==== 1415+0+0 (unknown 3020703955 0 0) 0x7fc6b8005c30 con 0x7fc6bc09ac40 2024-03-29T11:19:33.892 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.895+0000 7fc6b1ffb640 10 client.4138.objecter ms_dispatch 0x7fc6bc000b90 osd_map(8..8 src has 1..8) v4 2024-03-29T11:19:33.893 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.895+0000 7fc6b1ffb640 3 client.4138.objecter handle_osd_map got epochs [8,8] > 0 2024-03-29T11:19:33.893 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.895+0000 7fc6b1ffb640 3 client.4138.objecter handle_osd_map decoding full epoch 8 2024-03-29T11:19:33.893 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.895+0000 7fc6b1ffb640 20 client.4138.objecter dump_active .. 0 homeless 2024-03-29T11:19:33.893 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.896+0000 7fc6c36d9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:33.893 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.896+0000 7fc6c36d9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:33.893 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.896+0000 7fc6c36d9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-29T11:19:33.893 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.896+0000 7fc6c36d9640 1 -- 172.21.15.179:0/689965376 --> v1:172.21.15.179:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc6bc140380 con 0x7fc6bc09ac40 2024-03-29T11:19:33.893 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.898+0000 7fc6b1ffb640 1 -- 172.21.15.179:0/689965376 <== mon.1 v1:172.21.15.179:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7fc6bc140380 con 0x7fc6bc09ac40 2024-03-29T11:19:33.893 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.898+0000 7fc6b1ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:33.893 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:33.898+0000 7fc6b1ffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:19:33.977 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:34.001+0000 7fc6c36d9640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "d895eaa0-cc0b-4ee4-9b5a-2dfcd89a60b7", "id": 6}] 2024-03-29T11:19:33.978 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:34.001+0000 7fc6c36d9640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "d895eaa0-cc0b-4ee4-9b5a-2dfcd89a60b7", "id": 6}] 2024-03-29T11:19:33.978 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:34.001+0000 7fc6c36d9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-29T11:19:33.978 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:34.001+0000 7fc6c36d9640 1 -- 172.21.15.179:0/689965376 --> v1:172.21.15.179:6789/0 -- mon_command({"prefix": "osd new", "uuid": "d895eaa0-cc0b-4ee4-9b5a-2dfcd89a60b7", "id": 6} v 0) v1 -- 0x7fc6bc09b020 con 0x7fc6bc09ac40 2024-03-29T11:19:33.981 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:34.005+0000 7fc6b1ffb640 1 -- 172.21.15.179:0/689965376 <== mon.1 v1:172.21.15.179:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "d895eaa0-cc0b-4ee4-9b5a-2dfcd89a60b7", "id": 6}]=0 v9) v1 ==== 112+0+2 (unknown 742035447 0 1113811849) 0x7fc6b8005820 con 0x7fc6bc09ac40 2024-03-29T11:19:33.981 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:34.005+0000 7fc6b1ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "d895eaa0-cc0b-4ee4-9b5a-2dfcd89a60b7", "id": 6}] 2024-03-29T11:19:33.981 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:34.005+0000 7fc6b1ffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:19:33.982 INFO:teuthology.orchestra.run.smithi179.stdout:6 2024-03-29T11:19:33.983 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:34.005+0000 7fc6c36d9640 10 librados: watch_flush enter 2024-03-29T11:19:33.983 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:34.006+0000 7fc6c36d9640 10 librados: watch_flush exit 2024-03-29T11:19:33.983 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:34.006+0000 7fc6c36d9640 20 client.4138.objecter shutdown clearing up homeless session... 2024-03-29T11:19:33.983 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:34.006+0000 7fc6c36d9640 10 client.4138.objecter successfully canceled tick 2024-03-29T11:19:33.983 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:34.006+0000 7fc6c36d9640 10 monclient: shutdown 2024-03-29T11:19:33.983 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:34.006+0000 7fc6c36d9640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:19:33.983 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:34.006+0000 7fc6c36d9640 1 -- 172.21.15.179:0/689965376 >> v1:172.21.15.179:6789/0 conn(0x7fc6bc09ac40 legacy=0x7fc6bc142a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:33.983 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:34.006+0000 7fc6c36d9640 1 -- 172.21.15.179:0/689965376 shutdown_connections 2024-03-29T11:19:33.983 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:34.006+0000 7fc6c36d9640 1 -- 172.21.15.179:0/689965376 >> 172.21.15.179:0/689965376 conn(0x7fc6bc066b00 msgr2=0x7fc6bc13d7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:19:33.984 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:34.006+0000 7fc6c36d9640 1 -- 172.21.15.179:0/689965376 shutdown_connections 2024-03-29T11:19:33.984 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:34.006+0000 7fc6c36d9640 1 -- 172.21.15.179:0/689965376 wait complete. 2024-03-29T11:19:33.984 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:34.006+0000 7fc6c36d9640 1 librados: shutdown 2024-03-29T11:19:33.993 DEBUG:teuthology.orchestra.run.smithi179:> sudo ceph --cluster ceph osd new 9bc5a7bc-8f52-4db9-96d6-de5e68d52eaa 7 2024-03-29T11:19:34.009 INFO:tasks.ceph.mgr.y.smithi084.stderr:2024-03-29T11:19:34.033+0000 7f45957a8040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-29T11:19:34.094 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:34.117+0000 7f99d4d4b640 10 monclient: get_monmap_and_config 2024-03-29T11:19:34.094 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:34.117+0000 7f99d4d4b640 10 monclient: build_initial_monmap 2024-03-29T11:19:34.094 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:34.117+0000 7f99d4d4b640 10 monclient: monmap: 2024-03-29T11:19:34.094 INFO:teuthology.orchestra.run.smithi179.stderr:epoch 0 2024-03-29T11:19:34.094 INFO:teuthology.orchestra.run.smithi179.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:19:34.094 INFO:teuthology.orchestra.run.smithi179.stderr:last_changed 2024-03-29T11:19:34.118138+0000 2024-03-29T11:19:34.094 INFO:teuthology.orchestra.run.smithi179.stderr:created 2024-03-29T11:19:34.118138+0000 2024-03-29T11:19:34.094 INFO:teuthology.orchestra.run.smithi179.stderr:min_mon_release 0 (unknown) 2024-03-29T11:19:34.094 INFO:teuthology.orchestra.run.smithi179.stderr:election_strategy: 1 2024-03-29T11:19:34.094 INFO:teuthology.orchestra.run.smithi179.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-29T11:19:34.094 INFO:teuthology.orchestra.run.smithi179.stderr:1: v1:172.21.15.179:6789/0 mon.noname-c 2024-03-29T11:19:34.094 INFO:teuthology.orchestra.run.smithi179.stderr:2: v1:172.21.15.84:6790/0 mon.noname-b 2024-03-29T11:19:34.094 INFO:teuthology.orchestra.run.smithi179.stderr: 2024-03-29T11:19:34.102 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:34.117+0000 7f99d4d4b640 1 Processor -- start 2024-03-29T11:19:34.102 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:34.117+0000 7f99d4d4b640 1 -- start start 2024-03-29T11:19:34.102 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:34.117+0000 7f99d4d4b640 10 monclient: init 2024-03-29T11:19:34.103 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:34.118+0000 7f99d4d4b640 10 monclient: _reopen_session rank -1 2024-03-29T11:19:34.103 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:34.118+0000 7f99d4d4b640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:19:34.103 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:34.118+0000 7f99d4d4b640 10 monclient(hunting): picked mon.noname-c con 0x7f99d00b1780 addr v1:172.21.15.179:6789/0 2024-03-29T11:19:34.103 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:34.118+0000 7f99d4d4b640 10 monclient(hunting): picked mon.noname-b con 0x7f99d013d7e0 addr v1:172.21.15.84:6790/0 2024-03-29T11:19:34.103 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:34.118+0000 7f99d4d4b640 10 monclient(hunting): picked mon.noname-a con 0x7f99d0143be0 addr v1:172.21.15.84:6789/0 2024-03-29T11:19:34.103 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:34.118+0000 7f99d4d4b640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f99d0070b30 con 0x7f99d0143be0 2024-03-29T11:19:34.103 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:34.118+0000 7f99d4d4b640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f99d0070da0 con 0x7f99d00b1780 2024-03-29T11:19:34.103 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:34.118+0000 7f99d4d4b640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f99d009adb0 con 0x7f99d013d7e0 2024-03-29T11:19:34.103 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:34.118+0000 7f99d4d4b640 10 monclient(hunting): _renew_subs 2024-03-29T11:19:34.103 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:34.118+0000 7f99d4d4b640 10 monclient(hunting): authenticate will time out at 812.450660s 2024-03-29T11:19:34.103 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:34.118+0000 7f99ce59c640 1 --1- >> v1:172.21.15.179:6789/0 conn(0x7f99d00b1780 0x7f99d013d0d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.179:6789/0 says I am v1:172.21.15.179:53496/0 (socket says 172.21.15.179:53496) 2024-03-29T11:19:34.103 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:34.118+0000 7f99ce59c640 1 -- 172.21.15.179:0/3261976835 learned_addr learned my addr 172.21.15.179:0/3261976835 (peer_addr_for_me v1:172.21.15.179:0/0) 2024-03-29T11:19:34.103 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:34.118+0000 7f99cdd9b640 1 -- 172.21.15.179:0/3261976835 <== mon.1 v1:172.21.15.179:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2163936393 0 0) 0x7f99d0070da0 con 0x7f99d00b1780 2024-03-29T11:19:34.104 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:34.118+0000 7f99cdd9b640 10 monclient(hunting): _negotiate 2024-03-29T11:19:34.104 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:34.118+0000 7f99cdd9b640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:34.104 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:34.118+0000 7f99cdd9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:34.104 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:34.118+0000 7f99cdd9b640 10 monclient(hunting): my global_id is 4141 2024-03-29T11:19:34.104 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:34.118+0000 7f99cdd9b640 1 -- 172.21.15.179:0/3261976835 --> v1:172.21.15.179:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f99bc003520 con 0x7f99d00b1780 2024-03-29T11:19:34.104 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:34.119+0000 7f99cdd9b640 1 -- 172.21.15.179:0/3261976835 <== mon.0 v1:172.21.15.84:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1737099061 0 0) 0x7f99d0070b30 con 0x7f99d0143be0 2024-03-29T11:19:34.104 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:34.119+0000 7f99cdd9b640 10 monclient(hunting): _negotiate 2024-03-29T11:19:34.104 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:34.119+0000 7f99cdd9b640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:34.104 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:34.119+0000 7f99cdd9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:34.104 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:34.119+0000 7f99cdd9b640 10 monclient(hunting): my global_id is 4158 2024-03-29T11:19:34.104 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:34.119+0000 7f99cdd9b640 1 -- 172.21.15.179:0/3261976835 --> v1:172.21.15.84:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f99d0070da0 con 0x7f99d0143be0 2024-03-29T11:19:34.104 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:34.119+0000 7f99cdd9b640 1 -- 172.21.15.179:0/3261976835 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2688993019 0 0) 0x7f99d009adb0 con 0x7f99d013d7e0 2024-03-29T11:19:34.104 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:34.119+0000 7f99cdd9b640 10 monclient(hunting): _negotiate 2024-03-29T11:19:34.104 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:34.119+0000 7f99cdd9b640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:34.104 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:34.119+0000 7f99cdd9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:34.104 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:34.119+0000 7f99cdd9b640 10 monclient(hunting): my global_id is 4157 2024-03-29T11:19:34.105 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:34.119+0000 7f99cdd9b640 1 -- 172.21.15.179:0/3261976835 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f99d0070b30 con 0x7f99d013d7e0 2024-03-29T11:19:34.105 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:34.119+0000 7f99cdd9b640 1 -- 172.21.15.179:0/3261976835 <== mon.1 v1:172.21.15.179:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1719781149 0 0) 0x7f99bc003520 con 0x7f99d00b1780 2024-03-29T11:19:34.105 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:34.119+0000 7f99cdd9b640 1 -- 172.21.15.179:0/3261976835 >> v1:172.21.15.84:6790/0 conn(0x7f99d013d7e0 legacy=0x7f99d01434d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:34.105 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:34.119+0000 7f99cdd9b640 1 -- 172.21.15.179:0/3261976835 >> v1:172.21.15.84:6789/0 conn(0x7f99d0143be0 legacy=0x7f99d014b780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:34.105 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:34.119+0000 7f99cdd9b640 10 monclient: _finish_hunting 0 2024-03-29T11:19:34.105 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:34.119+0000 7f99cdd9b640 1 monclient: found mon.noname-c 2024-03-29T11:19:34.105 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:34.119+0000 7f99cdd9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:19:34.105 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:34.119+0000 7f99cdd9b640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.179:6789/0 2024-03-29T11:19:34.105 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:34.119+0000 7f99cdd9b640 1 -- 172.21.15.179:0/3261976835 --> v1:172.21.15.179:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99d0144070 con 0x7f99d00b1780 2024-03-29T11:19:34.105 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:34.119+0000 7f99cdd9b640 10 monclient: _finish_auth 0 2024-03-29T11:19:34.105 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:34.119+0000 7f99cdd9b640 10 monclient: _check_auth_tickets 2024-03-29T11:19:34.105 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:34.119+0000 7f99cdd9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:19:34.105 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:34.119+0000 7f99d4d4b640 5 monclient: authenticate success, global_id 4141 2024-03-29T11:19:34.105 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:34.119+0000 7f99cdd9b640 1 -- 172.21.15.179:0/3261976835 <== mon.1 v1:172.21.15.179:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4236868694 0 0) 0x7f99c0001690 con 0x7f99d00b1780 2024-03-29T11:19:34.105 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:34.119+0000 7f99d4d4b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:19:34.106 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:34.119+0000 7f99cdd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:34.106 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:34.119+0000 7f99cdd9b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:19:34.106 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:34.119+0000 7f99cdd9b640 10 monclient: dump: 2024-03-29T11:19:34.106 INFO:teuthology.orchestra.run.smithi179.stderr:epoch 1 2024-03-29T11:19:34.106 INFO:teuthology.orchestra.run.smithi179.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:19:34.106 INFO:teuthology.orchestra.run.smithi179.stderr:last_changed 2024-03-29T11:18:53.864025+0000 2024-03-29T11:19:34.106 INFO:teuthology.orchestra.run.smithi179.stderr:created 2024-03-29T11:18:53.864025+0000 2024-03-29T11:19:34.106 INFO:teuthology.orchestra.run.smithi179.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:34.106 INFO:teuthology.orchestra.run.smithi179.stderr:election_strategy: 1 2024-03-29T11:19:34.106 INFO:teuthology.orchestra.run.smithi179.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T11:19:34.106 INFO:teuthology.orchestra.run.smithi179.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T11:19:34.106 INFO:teuthology.orchestra.run.smithi179.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T11:19:34.106 INFO:teuthology.orchestra.run.smithi179.stderr: 2024-03-29T11:19:34.107 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:34.119+0000 7f99d4d4b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:19:34.107 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:34.119+0000 7f99cdd9b640 1 -- 172.21.15.179:0/3261976835 <== mon.1 v1:172.21.15.179:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f99c0001870 con 0x7f99d00b1780 2024-03-29T11:19:34.107 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:34.119+0000 7f99cdd9b640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:19:34.107 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:34.119+0000 7f99d4d4b640 10 monclient: get_monmap_and_config success 2024-03-29T11:19:34.107 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:34.119+0000 7f99d4d4b640 4 set_mon_vals no callback set 2024-03-29T11:19:34.107 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:34.119+0000 7f99cdd9b640 1 -- 172.21.15.179:0/3261976835 <== mon.1 v1:172.21.15.179:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4236868694 0 0) 0x7f99c0001c80 con 0x7f99d00b1780 2024-03-29T11:19:34.107 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:34.119+0000 7f99cdd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:34.107 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:34.119+0000 7f99cdd9b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:19:34.107 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:34.119+0000 7f99cdd9b640 10 monclient: dump: 2024-03-29T11:19:34.107 INFO:teuthology.orchestra.run.smithi179.stderr:epoch 1 2024-03-29T11:19:34.107 INFO:teuthology.orchestra.run.smithi179.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:19:34.107 INFO:teuthology.orchestra.run.smithi179.stderr:last_changed 2024-03-29T11:18:53.864025+0000 2024-03-29T11:19:34.107 INFO:teuthology.orchestra.run.smithi179.stderr:created 2024-03-29T11:18:53.864025+0000 2024-03-29T11:19:34.107 INFO:teuthology.orchestra.run.smithi179.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:34.107 INFO:teuthology.orchestra.run.smithi179.stderr:election_strategy: 1 2024-03-29T11:19:34.107 INFO:teuthology.orchestra.run.smithi179.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T11:19:34.108 INFO:teuthology.orchestra.run.smithi179.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T11:19:34.108 INFO:teuthology.orchestra.run.smithi179.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T11:19:34.108 INFO:teuthology.orchestra.run.smithi179.stderr: 2024-03-29T11:19:34.108 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:34.119+0000 7f99d4d4b640 10 monclient: shutdown 2024-03-29T11:19:34.108 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:34.119+0000 7f99d4d4b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:19:34.108 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:34.119+0000 7f99d4d4b640 1 -- 172.21.15.179:0/3261976835 >> v1:172.21.15.179:6789/0 conn(0x7f99d00b1780 legacy=0x7f99d013d0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:34.108 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:34.119+0000 7f99d4d4b640 1 -- 172.21.15.179:0/3261976835 shutdown_connections 2024-03-29T11:19:34.108 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:34.119+0000 7f99d4d4b640 1 -- 172.21.15.179:0/3261976835 >> 172.21.15.179:0/3261976835 conn(0x7f99d00b58f0 msgr2=0x7f99d00b5d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:19:34.108 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:34.119+0000 7f99d4d4b640 1 -- 172.21.15.179:0/3261976835 shutdown_connections 2024-03-29T11:19:34.108 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:34.119+0000 7f99d4d4b640 1 -- 172.21.15.179:0/3261976835 wait complete. 2024-03-29T11:19:34.108 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:34.120+0000 7f99d4d4b640 10 monclient: build_initial_monmap 2024-03-29T11:19:34.108 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:34.120+0000 7f99d4d4b640 10 monclient: monmap: 2024-03-29T11:19:34.108 INFO:teuthology.orchestra.run.smithi179.stderr:epoch 0 2024-03-29T11:19:34.108 INFO:teuthology.orchestra.run.smithi179.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:19:34.108 INFO:teuthology.orchestra.run.smithi179.stderr:last_changed 0.000000 2024-03-29T11:19:34.109 INFO:teuthology.orchestra.run.smithi179.stderr:created 0.000000 2024-03-29T11:19:34.109 INFO:teuthology.orchestra.run.smithi179.stderr:min_mon_release 0 (unknown) 2024-03-29T11:19:34.109 INFO:teuthology.orchestra.run.smithi179.stderr:election_strategy: 1 2024-03-29T11:19:34.109 INFO:teuthology.orchestra.run.smithi179.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-29T11:19:34.109 INFO:teuthology.orchestra.run.smithi179.stderr:1: v1:172.21.15.179:6789/0 mon.noname-b 2024-03-29T11:19:34.109 INFO:teuthology.orchestra.run.smithi179.stderr:2: v1:172.21.15.84:6790/0 mon.noname-c 2024-03-29T11:19:34.109 INFO:teuthology.orchestra.run.smithi179.stderr: 2024-03-29T11:19:34.109 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:34.120+0000 7f99d4d4b640 1 librados: starting msgr at 2024-03-29T11:19:34.109 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:34.120+0000 7f99d4d4b640 1 librados: starting objecter 2024-03-29T11:19:34.109 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:34.121+0000 7f99d4d4b640 1 Processor -- start 2024-03-29T11:19:34.109 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:34.121+0000 7f99d4d4b640 1 -- start start 2024-03-29T11:19:34.109 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:34.121+0000 7f99d4d4b640 1 librados: setting wanted keys 2024-03-29T11:19:34.109 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:34.121+0000 7f99d4d4b640 1 librados: calling monclient init 2024-03-29T11:19:34.109 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:34.121+0000 7f99d4d4b640 10 monclient: init 2024-03-29T11:19:34.109 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:34.121+0000 7f99d4d4b640 10 monclient: _reopen_session rank -1 2024-03-29T11:19:34.110 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:34.121+0000 7f99d4d4b640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:19:34.110 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:34.121+0000 7f99d4d4b640 10 monclient(hunting): picked mon.noname-c con 0x7f99d00b1780 addr v1:172.21.15.84:6790/0 2024-03-29T11:19:34.110 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:34.121+0000 7f99d4d4b640 10 monclient(hunting): picked mon.noname-b con 0x7f99d013d7e0 addr v1:172.21.15.179:6789/0 2024-03-29T11:19:34.110 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:34.121+0000 7f99d4d4b640 10 monclient(hunting): picked mon.noname-a con 0x7f99d0143be0 addr v1:172.21.15.84:6789/0 2024-03-29T11:19:34.110 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:34.121+0000 7f99d4d4b640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f99d0138d60 con 0x7f99d0143be0 2024-03-29T11:19:34.110 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:34.121+0000 7f99d4d4b640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f99d0135790 con 0x7f99d013d7e0 2024-03-29T11:19:34.110 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:34.121+0000 7f99d4d4b640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f99d0140c60 con 0x7f99d00b1780 2024-03-29T11:19:34.110 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:34.121+0000 7f99d4d4b640 10 monclient(hunting): _renew_subs 2024-03-29T11:19:34.110 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:34.121+0000 7f99d4d4b640 10 monclient(hunting): authenticate will time out at 812.453695s 2024-03-29T11:19:34.110 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:34.121+0000 7f99c7fff640 1 --1- >> v1:172.21.15.179:6789/0 conn(0x7f99d013d7e0 0x7f99d0134970 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.179:6789/0 says I am v1:172.21.15.179:53500/0 (socket says 172.21.15.179:53500) 2024-03-29T11:19:34.110 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:34.121+0000 7f99c7fff640 1 -- 172.21.15.179:0/4014243531 learned_addr learned my addr 172.21.15.179:0/4014243531 (peer_addr_for_me v1:172.21.15.179:0/0) 2024-03-29T11:19:34.110 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:34.121+0000 7f99c6ffd640 10 client.?.objecter ms_handle_connect 0x7f99d013d7e0 2024-03-29T11:19:34.110 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:34.121+0000 7f99c6ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:19:34.110 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:34.121+0000 7f99c6ffd640 10 client.?.objecter ms_handle_connect 0x7f99d00b1780 2024-03-29T11:19:34.110 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:34.121+0000 7f99c6ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:19:34.111 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:34.121+0000 7f99c6ffd640 10 client.?.objecter ms_handle_connect 0x7f99d0143be0 2024-03-29T11:19:34.111 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:34.121+0000 7f99c6ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:19:34.111 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:34.121+0000 7f99c6ffd640 1 -- 172.21.15.179:0/4014243531 <== mon.1 v1:172.21.15.179:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2018771728 0 0) 0x7f99d0135790 con 0x7f99d013d7e0 2024-03-29T11:19:34.111 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:34.121+0000 7f99c6ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:19:34.111 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:34.121+0000 7f99c6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:34.111 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:34.121+0000 7f99c6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:34.111 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:34.121+0000 7f99c6ffd640 10 monclient(hunting): my global_id is 4144 2024-03-29T11:19:34.111 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:34.121+0000 7f99c6ffd640 1 -- 172.21.15.179:0/4014243531 --> v1:172.21.15.179:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f99b0004eb0 con 0x7f99d013d7e0 2024-03-29T11:19:34.111 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:34.121+0000 7f99c6ffd640 1 -- 172.21.15.179:0/4014243531 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3583679345 0 0) 0x7f99d0140c60 con 0x7f99d00b1780 2024-03-29T11:19:34.111 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:34.121+0000 7f99c6ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:19:34.111 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:34.121+0000 7f99c6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:34.111 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:34.121+0000 7f99c6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:34.111 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:34.121+0000 7f99c6ffd640 10 monclient(hunting): my global_id is 4160 2024-03-29T11:19:34.111 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:34.121+0000 7f99c6ffd640 1 -- 172.21.15.179:0/4014243531 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f99d0135790 con 0x7f99d00b1780 2024-03-29T11:19:34.111 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:34.121+0000 7f99c6ffd640 1 -- 172.21.15.179:0/4014243531 <== mon.0 v1:172.21.15.84:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 727785751 0 0) 0x7f99d0138d60 con 0x7f99d0143be0 2024-03-29T11:19:34.112 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:34.122+0000 7f99c6ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:19:34.112 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:34.122+0000 7f99c6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:34.112 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:34.122+0000 7f99c6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:34.112 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:34.122+0000 7f99c6ffd640 10 monclient(hunting): my global_id is 4161 2024-03-29T11:19:34.112 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:34.122+0000 7f99c6ffd640 1 -- 172.21.15.179:0/4014243531 --> v1:172.21.15.84:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f99d0140c60 con 0x7f99d0143be0 2024-03-29T11:19:34.112 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:34.122+0000 7f99c6ffd640 1 -- 172.21.15.179:0/4014243531 <== mon.2 v1:172.21.15.84:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1149528628 0 0) 0x7f99d0135790 con 0x7f99d00b1780 2024-03-29T11:19:34.112 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:34.122+0000 7f99c6ffd640 1 -- 172.21.15.179:0/4014243531 --> v1:172.21.15.84:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f99d0138d60 con 0x7f99d00b1780 2024-03-29T11:19:34.112 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:34.122+0000 7f99c6ffd640 1 -- 172.21.15.179:0/4014243531 <== mon.1 v1:172.21.15.179:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1252692987 0 0) 0x7f99b0004eb0 con 0x7f99d013d7e0 2024-03-29T11:19:34.112 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:34.122+0000 7f99c6ffd640 1 -- 172.21.15.179:0/4014243531 --> v1:172.21.15.179:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f99d0135790 con 0x7f99d013d7e0 2024-03-29T11:19:34.112 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:34.122+0000 7f99c6ffd640 1 -- 172.21.15.179:0/4014243531 <== mon.2 v1:172.21.15.84:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4236868694 0 0) 0x7f99c0001b80 con 0x7f99d00b1780 2024-03-29T11:19:34.112 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:34.122+0000 7f99c6ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:34.112 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:34.122+0000 7f99c6ffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:19:34.112 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:34.122+0000 7f99c6ffd640 10 monclient(hunting): dump: 2024-03-29T11:19:34.112 INFO:teuthology.orchestra.run.smithi179.stderr:epoch 1 2024-03-29T11:19:34.113 INFO:teuthology.orchestra.run.smithi179.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:19:34.113 INFO:teuthology.orchestra.run.smithi179.stderr:last_changed 2024-03-29T11:18:53.864025+0000 2024-03-29T11:19:34.113 INFO:teuthology.orchestra.run.smithi179.stderr:created 2024-03-29T11:18:53.864025+0000 2024-03-29T11:19:34.113 INFO:teuthology.orchestra.run.smithi179.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:34.113 INFO:teuthology.orchestra.run.smithi179.stderr:election_strategy: 1 2024-03-29T11:19:34.113 INFO:teuthology.orchestra.run.smithi179.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T11:19:34.113 INFO:teuthology.orchestra.run.smithi179.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T11:19:34.113 INFO:teuthology.orchestra.run.smithi179.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T11:19:34.113 INFO:teuthology.orchestra.run.smithi179.stderr: 2024-03-29T11:19:34.113 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:34.122+0000 7f99c6ffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:19:34.113 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:34.122+0000 7f99c6ffd640 1 -- 172.21.15.179:0/4014243531 <== mon.1 v1:172.21.15.179:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4236868694 0 0) 0x7f99b8001a10 con 0x7f99d013d7e0 2024-03-29T11:19:34.113 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:34.122+0000 7f99c6ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:34.113 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:34.122+0000 7f99c6ffd640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:19:34.113 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:34.122+0000 7f99c6ffd640 10 monclient(hunting): dump: 2024-03-29T11:19:34.113 INFO:teuthology.orchestra.run.smithi179.stderr:epoch 1 2024-03-29T11:19:34.113 INFO:teuthology.orchestra.run.smithi179.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:19:34.114 INFO:teuthology.orchestra.run.smithi179.stderr:last_changed 2024-03-29T11:18:53.864025+0000 2024-03-29T11:19:34.114 INFO:teuthology.orchestra.run.smithi179.stderr:created 2024-03-29T11:18:53.864025+0000 2024-03-29T11:19:34.114 INFO:teuthology.orchestra.run.smithi179.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:34.114 INFO:teuthology.orchestra.run.smithi179.stderr:election_strategy: 1 2024-03-29T11:19:34.114 INFO:teuthology.orchestra.run.smithi179.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T11:19:34.114 INFO:teuthology.orchestra.run.smithi179.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T11:19:34.114 INFO:teuthology.orchestra.run.smithi179.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T11:19:34.114 INFO:teuthology.orchestra.run.smithi179.stderr: 2024-03-29T11:19:34.114 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:34.122+0000 7f99c6ffd640 1 -- 172.21.15.179:0/4014243531 <== mon.0 v1:172.21.15.84:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 673268947 0 0) 0x7f99d0140c60 con 0x7f99d0143be0 2024-03-29T11:19:34.114 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:34.122+0000 7f99c6ffd640 1 -- 172.21.15.179:0/4014243531 --> v1:172.21.15.84:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f99b0004eb0 con 0x7f99d0143be0 2024-03-29T11:19:34.114 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:34.122+0000 7f99c6ffd640 1 -- 172.21.15.179:0/4014243531 <== mon.0 v1:172.21.15.84:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4236868694 0 0) 0x7f99c8001a30 con 0x7f99d0143be0 2024-03-29T11:19:34.114 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:34.122+0000 7f99c6ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:34.114 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:34.122+0000 7f99c6ffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:19:34.114 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:34.122+0000 7f99c6ffd640 10 monclient(hunting): dump: 2024-03-29T11:19:34.114 INFO:teuthology.orchestra.run.smithi179.stderr:epoch 1 2024-03-29T11:19:34.115 INFO:teuthology.orchestra.run.smithi179.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:19:34.115 INFO:teuthology.orchestra.run.smithi179.stderr:last_changed 2024-03-29T11:18:53.864025+0000 2024-03-29T11:19:34.115 INFO:teuthology.orchestra.run.smithi179.stderr:created 2024-03-29T11:18:53.864025+0000 2024-03-29T11:19:34.115 INFO:teuthology.orchestra.run.smithi179.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:34.115 INFO:teuthology.orchestra.run.smithi179.stderr:election_strategy: 1 2024-03-29T11:19:34.115 INFO:teuthology.orchestra.run.smithi179.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T11:19:34.115 INFO:teuthology.orchestra.run.smithi179.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T11:19:34.115 INFO:teuthology.orchestra.run.smithi179.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T11:19:34.115 INFO:teuthology.orchestra.run.smithi179.stderr: 2024-03-29T11:19:34.115 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:34.122+0000 7f99c6ffd640 1 -- 172.21.15.179:0/4014243531 <== mon.2 v1:172.21.15.84:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4211136379 0 0) 0x7f99d0138d60 con 0x7f99d00b1780 2024-03-29T11:19:34.115 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:34.122+0000 7f99c6ffd640 1 -- 172.21.15.179:0/4014243531 >> v1:172.21.15.179:6789/0 conn(0x7f99d013d7e0 legacy=0x7f99d0134970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:34.115 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:34.122+0000 7f99c6ffd640 1 -- 172.21.15.179:0/4014243531 >> v1:172.21.15.84:6789/0 conn(0x7f99d0143be0 legacy=0x7f99d0135080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:34.115 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:34.122+0000 7f99c6ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:19:34.115 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:34.122+0000 7f99c6ffd640 1 monclient: found mon.c 2024-03-29T11:19:34.116 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:34.122+0000 7f99c6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:19:34.116 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:34.122+0000 7f99c6ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-29T11:19:34.116 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:34.122+0000 7f99c6ffd640 1 -- 172.21.15.179:0/4014243531 --> v1:172.21.15.84:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99d013a540 con 0x7f99d00b1780 2024-03-29T11:19:34.116 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:34.122+0000 7f99c6ffd640 10 monclient: _finish_auth 0 2024-03-29T11:19:34.116 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:34.122+0000 7f99c6ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:19:34.116 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:34.122+0000 7f99c6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:19:34.116 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:34.122+0000 7f99d4d4b640 5 monclient: authenticate success, global_id 4160 2024-03-29T11:19:34.116 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:34.122+0000 7f99d4d4b640 10 monclient: _renew_subs 2024-03-29T11:19:34.116 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:34.122+0000 7f99d4d4b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-29T11:19:34.116 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:34.122+0000 7f99d4d4b640 1 -- 172.21.15.179:0/4014243531 --> v1:172.21.15.84:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f99d013f5a0 con 0x7f99d00b1780 2024-03-29T11:19:34.116 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:34.122+0000 7f99d4d4b640 10 client.4160.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:19:34.116 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:34.122+0000 7f99d4d4b640 10 monclient: _renew_subs 2024-03-29T11:19:34.116 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:34.122+0000 7f99d4d4b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-29T11:19:34.116 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:34.122+0000 7f99d4d4b640 1 -- 172.21.15.179:0/4014243531 --> v1:172.21.15.84:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f99d01393d0 con 0x7f99d00b1780 2024-03-29T11:19:34.116 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:34.122+0000 7f99d4d4b640 1 librados: init done 2024-03-29T11:19:34.116 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:34.122+0000 7f99c6ffd640 1 -- 172.21.15.179:0/4014243531 <== mon.2 v1:172.21.15.84:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f99c0004bf0 con 0x7f99d00b1780 2024-03-29T11:19:34.117 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:34.122+0000 7f99c6ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:19:34.117 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:34.122+0000 7f99c6ffd640 1 -- 172.21.15.179:0/4014243531 <== mon.2 v1:172.21.15.84:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4236868694 0 0) 0x7f99c0004f00 con 0x7f99d00b1780 2024-03-29T11:19:34.117 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:34.122+0000 7f99c6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:34.117 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:34.122+0000 7f99c6ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:19:34.117 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:34.122+0000 7f99cd59a640 4 set_mon_vals no callback set 2024-03-29T11:19:34.117 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:34.122+0000 7f99c6ffd640 10 monclient: dump: 2024-03-29T11:19:34.117 INFO:teuthology.orchestra.run.smithi179.stderr:epoch 1 2024-03-29T11:19:34.117 INFO:teuthology.orchestra.run.smithi179.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:19:34.117 INFO:teuthology.orchestra.run.smithi179.stderr:last_changed 2024-03-29T11:18:53.864025+0000 2024-03-29T11:19:34.117 INFO:teuthology.orchestra.run.smithi179.stderr:created 2024-03-29T11:18:53.864025+0000 2024-03-29T11:19:34.117 INFO:teuthology.orchestra.run.smithi179.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:34.117 INFO:teuthology.orchestra.run.smithi179.stderr:election_strategy: 1 2024-03-29T11:19:34.117 INFO:teuthology.orchestra.run.smithi179.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T11:19:34.117 INFO:teuthology.orchestra.run.smithi179.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T11:19:34.117 INFO:teuthology.orchestra.run.smithi179.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T11:19:34.117 INFO:teuthology.orchestra.run.smithi179.stderr: 2024-03-29T11:19:34.118 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:34.122+0000 7f99c6ffd640 1 -- 172.21.15.179:0/4014243531 <== mon.2 v1:172.21.15.84:6790/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f99c00053f0 con 0x7f99d00b1780 2024-03-29T11:19:34.118 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:34.122+0000 7f99c6ffd640 1 -- 172.21.15.179:0/4014243531 <== mon.2 v1:172.21.15.84:6790/0 8 ==== osd_map(9..9 src has 1..9) v4 ==== 1530+0+0 (unknown 1256302828 0 0) 0x7f99c0006aa0 con 0x7f99d00b1780 2024-03-29T11:19:34.118 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:34.122+0000 7f99c6ffd640 10 client.4160.objecter ms_dispatch 0x7f99d0000b90 osd_map(9..9 src has 1..9) v4 2024-03-29T11:19:34.118 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:34.122+0000 7f99c6ffd640 3 client.4160.objecter handle_osd_map got epochs [9,9] > 0 2024-03-29T11:19:34.118 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:34.122+0000 7f99c6ffd640 3 client.4160.objecter handle_osd_map decoding full epoch 9 2024-03-29T11:19:34.118 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:34.122+0000 7f99c6ffd640 20 client.4160.objecter dump_active .. 0 homeless 2024-03-29T11:19:34.118 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:34.123+0000 7f99d4d4b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:34.118 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:34.123+0000 7f99d4d4b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:34.118 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:34.123+0000 7f99d4d4b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-29T11:19:34.118 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:34.123+0000 7f99d4d4b640 1 -- 172.21.15.179:0/4014243531 --> v1:172.21.15.84:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f99b4005080 con 0x7f99d00b1780 2024-03-29T11:19:34.118 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:34.125+0000 7f99c6ffd640 1 -- 172.21.15.179:0/4014243531 <== mon.2 v1:172.21.15.84:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f99c0006440 con 0x7f99d00b1780 2024-03-29T11:19:34.118 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:34.125+0000 7f99c6ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:34.118 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:34.125+0000 7f99c6ffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:19:34.144 INFO:tasks.ceph.mgr.y.smithi084.stderr:2024-03-29T11:19:34.168+0000 7f45957a8040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-29T11:19:34.149 INFO:tasks.ceph.mgr.x.smithi179.stderr:2024-03-29T11:19:34.172+0000 7f8706237040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-29T11:19:34.175 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:34.198+0000 7f99d4d4b640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "9bc5a7bc-8f52-4db9-96d6-de5e68d52eaa", "id": 7}] 2024-03-29T11:19:34.175 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:34.198+0000 7f99d4d4b640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "9bc5a7bc-8f52-4db9-96d6-de5e68d52eaa", "id": 7}] 2024-03-29T11:19:34.175 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:34.198+0000 7f99d4d4b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-29T11:19:34.175 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:34.198+0000 7f99d4d4b640 1 -- 172.21.15.179:0/4014243531 --> v1:172.21.15.84:6790/0 -- mon_command({"prefix": "osd new", "uuid": "9bc5a7bc-8f52-4db9-96d6-de5e68d52eaa", "id": 7} v 0) v1 -- 0x7f99b40058b0 con 0x7f99d00b1780 2024-03-29T11:19:34.181 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:34.204+0000 7f99c6ffd640 1 -- 172.21.15.179:0/4014243531 <== mon.2 v1:172.21.15.84:6790/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "9bc5a7bc-8f52-4db9-96d6-de5e68d52eaa", "id": 7}]=0 v10) v1 ==== 112+0+2 (unknown 3391790811 0 1371665406) 0x7f99c00067b0 con 0x7f99d00b1780 2024-03-29T11:19:34.181 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:34.204+0000 7f99c6ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "9bc5a7bc-8f52-4db9-96d6-de5e68d52eaa", "id": 7}] 2024-03-29T11:19:34.181 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:34.204+0000 7f99c6ffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:19:34.181 INFO:teuthology.orchestra.run.smithi179.stdout:7 2024-03-29T11:19:34.183 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:34.205+0000 7f99d4d4b640 10 librados: watch_flush enter 2024-03-29T11:19:34.183 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:34.205+0000 7f99d4d4b640 10 librados: watch_flush exit 2024-03-29T11:19:34.183 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:34.205+0000 7f99d4d4b640 20 client.4160.objecter shutdown clearing up homeless session... 2024-03-29T11:19:34.183 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:34.205+0000 7f99d4d4b640 10 client.4160.objecter successfully canceled tick 2024-03-29T11:19:34.183 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:34.205+0000 7f99d4d4b640 10 monclient: shutdown 2024-03-29T11:19:34.183 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:34.205+0000 7f99d4d4b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:19:34.183 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:34.205+0000 7f99d4d4b640 1 -- 172.21.15.179:0/4014243531 >> v1:172.21.15.84:6790/0 conn(0x7f99d00b1780 legacy=0x7f99d013ee90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:34.183 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:34.205+0000 7f99d4d4b640 1 -- 172.21.15.179:0/4014243531 shutdown_connections 2024-03-29T11:19:34.183 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:34.205+0000 7f99d4d4b640 1 -- 172.21.15.179:0/4014243531 >> 172.21.15.179:0/4014243531 conn(0x7f99d00b58f0 msgr2=0x7f99d0069bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:19:34.183 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:34.205+0000 7f99d4d4b640 1 -- 172.21.15.179:0/4014243531 shutdown_connections 2024-03-29T11:19:34.183 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:34.205+0000 7f99d4d4b640 1 -- 172.21.15.179:0/4014243531 wait complete. 2024-03-29T11:19:34.183 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T11:19:34.205+0000 7f99d4d4b640 1 librados: shutdown 2024-03-29T11:19:34.192 INFO:tasks.ceph.osd.0:Restarting daemon 2024-03-29T11:19:34.192 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2024-03-29T11:19:34.194 INFO:tasks.ceph.osd.0:Started 2024-03-29T11:19:34.195 INFO:tasks.ceph.osd.1:Restarting daemon 2024-03-29T11:19:34.195 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2024-03-29T11:19:34.197 INFO:tasks.ceph.osd.1:Started 2024-03-29T11:19:34.197 INFO:tasks.ceph.osd.2:Restarting daemon 2024-03-29T11:19:34.197 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2024-03-29T11:19:34.199 INFO:tasks.ceph.osd.2:Started 2024-03-29T11:19:34.199 INFO:tasks.ceph.osd.3:Restarting daemon 2024-03-29T11:19:34.199 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2024-03-29T11:19:34.201 INFO:tasks.ceph.osd.3:Started 2024-03-29T11:19:34.201 INFO:tasks.ceph.osd.4:Restarting daemon 2024-03-29T11:19:34.201 DEBUG:teuthology.orchestra.run.smithi179:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 4 2024-03-29T11:19:34.203 INFO:tasks.ceph.osd.4:Started 2024-03-29T11:19:34.204 INFO:tasks.ceph.osd.5:Restarting daemon 2024-03-29T11:19:34.204 DEBUG:teuthology.orchestra.run.smithi179:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 5 2024-03-29T11:19:34.206 INFO:tasks.ceph.osd.5:Started 2024-03-29T11:19:34.206 INFO:tasks.ceph.osd.6:Restarting daemon 2024-03-29T11:19:34.206 DEBUG:teuthology.orchestra.run.smithi179:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 6 2024-03-29T11:19:34.208 INFO:tasks.ceph.osd.6:Started 2024-03-29T11:19:34.208 INFO:tasks.ceph.osd.7:Restarting daemon 2024-03-29T11:19:34.208 DEBUG:teuthology.orchestra.run.smithi179:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 7 2024-03-29T11:19:34.210 INFO:tasks.ceph.osd.7:Started 2024-03-29T11:19:34.211 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-29T11:19:34.303 INFO:tasks.ceph.mgr.x.smithi179.stderr:2024-03-29T11:19:34.326+0000 7f8706237040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-29T11:19:34.319 INFO:tasks.ceph.mgr.y.smithi084.stderr:2024-03-29T11:19:34.343+0000 7f45957a8040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-29T11:19:34.333 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T11:19:34.356+0000 7f86062c8640 10 monclient: get_monmap_and_config 2024-03-29T11:19:34.333 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T11:19:34.356+0000 7f86062c8640 10 monclient: build_initial_monmap 2024-03-29T11:19:34.333 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T11:19:34.356+0000 7f86062c8640 10 monclient: monmap: 2024-03-29T11:19:34.333 INFO:teuthology.orchestra.run.smithi084.stderr:epoch 0 2024-03-29T11:19:34.333 INFO:teuthology.orchestra.run.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:19:34.333 INFO:teuthology.orchestra.run.smithi084.stderr:last_changed 2024-03-29T11:19:34.357600+0000 2024-03-29T11:19:34.333 INFO:teuthology.orchestra.run.smithi084.stderr:created 2024-03-29T11:19:34.357600+0000 2024-03-29T11:19:34.334 INFO:teuthology.orchestra.run.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-29T11:19:34.334 INFO:teuthology.orchestra.run.smithi084.stderr:election_strategy: 1 2024-03-29T11:19:34.334 INFO:teuthology.orchestra.run.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-29T11:19:34.334 INFO:teuthology.orchestra.run.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.noname-c 2024-03-29T11:19:34.334 INFO:teuthology.orchestra.run.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-b 2024-03-29T11:19:34.334 INFO:teuthology.orchestra.run.smithi084.stderr: 2024-03-29T11:19:34.341 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T11:19:34.357+0000 7f86062c8640 1 Processor -- start 2024-03-29T11:19:34.341 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T11:19:34.357+0000 7f86062c8640 1 -- start start 2024-03-29T11:19:34.341 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T11:19:34.357+0000 7f86062c8640 10 monclient: init 2024-03-29T11:19:34.341 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T11:19:34.357+0000 7f86062c8640 10 monclient: _reopen_session rank -1 2024-03-29T11:19:34.341 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T11:19:34.357+0000 7f86062c8640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:19:34.342 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T11:19:34.358+0000 7f86062c8640 10 monclient(hunting): picked mon.noname-c con 0x7f86000b8240 addr v1:172.21.15.179:6789/0 2024-03-29T11:19:34.342 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T11:19:34.358+0000 7f86062c8640 10 monclient(hunting): picked mon.noname-a con 0x7f86000b1f60 addr v1:172.21.15.84:6789/0 2024-03-29T11:19:34.342 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T11:19:34.358+0000 7f86062c8640 10 monclient(hunting): picked mon.noname-b con 0x7f860009c8c0 addr v1:172.21.15.84:6790/0 2024-03-29T11:19:34.342 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T11:19:34.358+0000 7f86062c8640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f860009bd80 con 0x7f86000b1f60 2024-03-29T11:19:34.342 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T11:19:34.358+0000 7f86062c8640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f860009b960 con 0x7f86000b8240 2024-03-29T11:19:34.342 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T11:19:34.358+0000 7f86062c8640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f86000b2ec0 con 0x7f860009c8c0 2024-03-29T11:19:34.342 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T11:19:34.358+0000 7f86062c8640 10 monclient(hunting): _renew_subs 2024-03-29T11:19:34.342 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T11:19:34.358+0000 7f86062c8640 10 monclient(hunting): authenticate will time out at 835.652167s 2024-03-29T11:19:34.342 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T11:19:34.358+0000 7f85ff7fe640 1 --1- >> v1:172.21.15.84:6789/0 conn(0x7f86000b1f60 0x7f860009c1b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.84:6789/0 says I am v1:172.21.15.84:39060/0 (socket says 172.21.15.84:39060) 2024-03-29T11:19:34.343 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T11:19:34.358+0000 7f85ff7fe640 1 -- 172.21.15.84:0/3594763139 learned_addr learned my addr 172.21.15.84:0/3594763139 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-29T11:19:34.343 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T11:19:34.358+0000 7f85feffd640 1 -- 172.21.15.84:0/3594763139 <== mon.0 v1:172.21.15.84:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2988932611 0 0) 0x7f860009bd80 con 0x7f86000b1f60 2024-03-29T11:19:34.343 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T11:19:34.358+0000 7f85feffd640 10 monclient(hunting): _negotiate 2024-03-29T11:19:34.343 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T11:19:34.358+0000 7f85feffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:34.343 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T11:19:34.358+0000 7f85feffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:34.343 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T11:19:34.358+0000 7f85feffd640 10 monclient(hunting): my global_id is 4188 2024-03-29T11:19:34.344 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T11:19:34.358+0000 7f85feffd640 1 -- 172.21.15.84:0/3594763139 --> v1:172.21.15.84:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f85e8003520 con 0x7f86000b1f60 2024-03-29T11:19:34.344 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T11:19:34.358+0000 7f85feffd640 1 -- 172.21.15.84:0/3594763139 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1294463410 0 0) 0x7f86000b2ec0 con 0x7f860009c8c0 2024-03-29T11:19:34.344 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T11:19:34.358+0000 7f85feffd640 10 monclient(hunting): _negotiate 2024-03-29T11:19:34.344 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T11:19:34.358+0000 7f85feffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:34.344 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T11:19:34.358+0000 7f85feffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:34.344 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T11:19:34.358+0000 7f85feffd640 10 monclient(hunting): my global_id is 4187 2024-03-29T11:19:34.344 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T11:19:34.358+0000 7f85feffd640 1 -- 172.21.15.84:0/3594763139 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f860009bd80 con 0x7f860009c8c0 2024-03-29T11:19:34.344 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T11:19:34.358+0000 7f85feffd640 1 -- 172.21.15.84:0/3594763139 <== mon.1 v1:172.21.15.179:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3430224581 0 0) 0x7f860009b960 con 0x7f86000b8240 2024-03-29T11:19:34.344 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T11:19:34.358+0000 7f85feffd640 10 monclient(hunting): _negotiate 2024-03-29T11:19:34.344 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T11:19:34.358+0000 7f85feffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:34.344 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T11:19:34.358+0000 7f85feffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:34.344 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T11:19:34.358+0000 7f85feffd640 10 monclient(hunting): my global_id is 4171 2024-03-29T11:19:34.344 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T11:19:34.358+0000 7f85feffd640 1 -- 172.21.15.84:0/3594763139 --> v1:172.21.15.179:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f86000b2ec0 con 0x7f86000b8240 2024-03-29T11:19:34.345 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T11:19:34.358+0000 7f85feffd640 1 -- 172.21.15.84:0/3594763139 <== mon.0 v1:172.21.15.84:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2592030475 0 0) 0x7f85e8003520 con 0x7f86000b1f60 2024-03-29T11:19:34.345 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T11:19:34.358+0000 7f85feffd640 1 -- 172.21.15.84:0/3594763139 >> v1:172.21.15.84:6790/0 conn(0x7f860009c8c0 legacy=0x7f860014d900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:34.345 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T11:19:34.358+0000 7f85feffd640 1 -- 172.21.15.84:0/3594763139 >> v1:172.21.15.179:6789/0 conn(0x7f86000b8240 legacy=0x7f860013f450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:34.345 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T11:19:34.358+0000 7f85feffd640 10 monclient: _finish_hunting 0 2024-03-29T11:19:34.345 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T11:19:34.358+0000 7f85feffd640 1 monclient: found mon.noname-a 2024-03-29T11:19:34.345 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T11:19:34.358+0000 7f85feffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:19:34.345 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T11:19:34.358+0000 7f85feffd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.84:6789/0 2024-03-29T11:19:34.345 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T11:19:34.358+0000 7f85feffd640 1 -- 172.21.15.84:0/3594763139 --> v1:172.21.15.84:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f860013fb60 con 0x7f86000b1f60 2024-03-29T11:19:34.345 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T11:19:34.358+0000 7f85feffd640 10 monclient: _finish_auth 0 2024-03-29T11:19:34.345 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T11:19:34.358+0000 7f85feffd640 10 monclient: _check_auth_tickets 2024-03-29T11:19:34.345 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T11:19:34.358+0000 7f85feffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:19:34.345 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T11:19:34.358+0000 7f85feffd640 1 -- 172.21.15.84:0/3594763139 <== mon.0 v1:172.21.15.84:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4236868694 0 0) 0x7f85f0001690 con 0x7f86000b1f60 2024-03-29T11:19:34.345 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T11:19:34.358+0000 7f85feffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:34.345 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T11:19:34.358+0000 7f85feffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:19:34.345 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T11:19:34.358+0000 7f85feffd640 10 monclient: dump: 2024-03-29T11:19:34.345 INFO:teuthology.orchestra.run.smithi084.stderr:epoch 1 2024-03-29T11:19:34.346 INFO:teuthology.orchestra.run.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:19:34.346 INFO:teuthology.orchestra.run.smithi084.stderr:last_changed 2024-03-29T11:18:53.864025+0000 2024-03-29T11:19:34.346 INFO:teuthology.orchestra.run.smithi084.stderr:created 2024-03-29T11:18:53.864025+0000 2024-03-29T11:19:34.346 INFO:teuthology.orchestra.run.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:34.346 INFO:teuthology.orchestra.run.smithi084.stderr:election_strategy: 1 2024-03-29T11:19:34.346 INFO:teuthology.orchestra.run.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T11:19:34.346 INFO:teuthology.orchestra.run.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T11:19:34.346 INFO:teuthology.orchestra.run.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T11:19:34.346 INFO:teuthology.orchestra.run.smithi084.stderr: 2024-03-29T11:19:34.346 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T11:19:34.358+0000 7f86062c8640 5 monclient: authenticate success, global_id 4188 2024-03-29T11:19:34.346 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T11:19:34.358+0000 7f86062c8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:19:34.346 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T11:19:34.358+0000 7f85feffd640 1 -- 172.21.15.84:0/3594763139 <== mon.0 v1:172.21.15.84:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f85f0001870 con 0x7f86000b1f60 2024-03-29T11:19:34.346 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T11:19:34.358+0000 7f85feffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:19:34.346 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T11:19:34.358+0000 7f85feffd640 1 -- 172.21.15.84:0/3594763139 <== mon.0 v1:172.21.15.84:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4236868694 0 0) 0x7f85f0001cc0 con 0x7f86000b1f60 2024-03-29T11:19:34.347 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T11:19:34.358+0000 7f85feffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:34.347 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T11:19:34.358+0000 7f85feffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:19:34.347 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T11:19:34.358+0000 7f85feffd640 10 monclient: dump: 2024-03-29T11:19:34.347 INFO:teuthology.orchestra.run.smithi084.stderr:epoch 1 2024-03-29T11:19:34.347 INFO:teuthology.orchestra.run.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:19:34.347 INFO:teuthology.orchestra.run.smithi084.stderr:last_changed 2024-03-29T11:18:53.864025+0000 2024-03-29T11:19:34.347 INFO:teuthology.orchestra.run.smithi084.stderr:created 2024-03-29T11:18:53.864025+0000 2024-03-29T11:19:34.347 INFO:teuthology.orchestra.run.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:34.347 INFO:teuthology.orchestra.run.smithi084.stderr:election_strategy: 1 2024-03-29T11:19:34.347 INFO:teuthology.orchestra.run.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T11:19:34.347 INFO:teuthology.orchestra.run.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T11:19:34.347 INFO:teuthology.orchestra.run.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T11:19:34.347 INFO:teuthology.orchestra.run.smithi084.stderr: 2024-03-29T11:19:34.347 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T11:19:34.358+0000 7f86062c8640 10 monclient: get_monmap_and_config success 2024-03-29T11:19:34.348 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T11:19:34.358+0000 7f86062c8640 4 set_mon_vals no callback set 2024-03-29T11:19:34.348 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T11:19:34.359+0000 7f86062c8640 10 monclient: shutdown 2024-03-29T11:19:34.348 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T11:19:34.359+0000 7f86062c8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:19:34.348 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T11:19:34.359+0000 7f86062c8640 1 -- 172.21.15.84:0/3594763139 >> v1:172.21.15.84:6789/0 conn(0x7f86000b1f60 legacy=0x7f860009c1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:34.348 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T11:19:34.359+0000 7f86062c8640 1 -- 172.21.15.84:0/3594763139 shutdown_connections 2024-03-29T11:19:34.348 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T11:19:34.359+0000 7f86062c8640 1 -- 172.21.15.84:0/3594763139 >> 172.21.15.84:0/3594763139 conn(0x7f8600066b00 msgr2=0x7f8600066f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:19:34.348 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T11:19:34.359+0000 7f86062c8640 1 -- 172.21.15.84:0/3594763139 shutdown_connections 2024-03-29T11:19:34.348 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T11:19:34.359+0000 7f86062c8640 1 -- 172.21.15.84:0/3594763139 wait complete. 2024-03-29T11:19:34.348 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T11:19:34.360+0000 7f86062c8640 10 monclient: build_initial_monmap 2024-03-29T11:19:34.348 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T11:19:34.360+0000 7f86062c8640 10 monclient: monmap: 2024-03-29T11:19:34.348 INFO:teuthology.orchestra.run.smithi084.stderr:epoch 0 2024-03-29T11:19:34.348 INFO:teuthology.orchestra.run.smithi084.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:19:34.349 INFO:teuthology.orchestra.run.smithi084.stderr:last_changed 0.000000 2024-03-29T11:19:34.349 INFO:teuthology.orchestra.run.smithi084.stderr:created 0.000000 2024-03-29T11:19:34.349 INFO:teuthology.orchestra.run.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-29T11:19:34.349 INFO:teuthology.orchestra.run.smithi084.stderr:election_strategy: 1 2024-03-29T11:19:34.349 INFO:teuthology.orchestra.run.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-29T11:19:34.349 INFO:teuthology.orchestra.run.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.noname-b 2024-03-29T11:19:34.349 INFO:teuthology.orchestra.run.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-c 2024-03-29T11:19:34.349 INFO:teuthology.orchestra.run.smithi084.stderr: 2024-03-29T11:19:34.349 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T11:19:34.360+0000 7f86062c8640 1 librados: starting msgr at 2024-03-29T11:19:34.349 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T11:19:34.360+0000 7f86062c8640 1 librados: starting objecter 2024-03-29T11:19:34.349 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T11:19:34.360+0000 7f86062c8640 1 Processor -- start 2024-03-29T11:19:34.349 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T11:19:34.360+0000 7f86062c8640 1 -- start start 2024-03-29T11:19:34.350 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T11:19:34.360+0000 7f86062c8640 1 librados: setting wanted keys 2024-03-29T11:19:34.350 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T11:19:34.360+0000 7f86062c8640 1 librados: calling monclient init 2024-03-29T11:19:34.350 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T11:19:34.360+0000 7f86062c8640 10 monclient: init 2024-03-29T11:19:34.350 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T11:19:34.361+0000 7f86062c8640 10 monclient: _reopen_session rank -1 2024-03-29T11:19:34.350 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T11:19:34.361+0000 7f86062c8640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:19:34.350 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T11:19:34.361+0000 7f86062c8640 10 monclient(hunting): picked mon.noname-b con 0x7f860009c8c0 addr v1:172.21.15.179:6789/0 2024-03-29T11:19:34.350 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T11:19:34.361+0000 7f86062c8640 10 monclient(hunting): picked mon.noname-c con 0x7f86000b8240 addr v1:172.21.15.84:6790/0 2024-03-29T11:19:34.350 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T11:19:34.361+0000 7f86062c8640 10 monclient(hunting): picked mon.noname-a con 0x7f8600136410 addr v1:172.21.15.84:6789/0 2024-03-29T11:19:34.350 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T11:19:34.361+0000 7f86062c8640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8600137f80 con 0x7f8600136410 2024-03-29T11:19:34.350 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T11:19:34.361+0000 7f86062c8640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8600135c30 con 0x7f860009c8c0 2024-03-29T11:19:34.350 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T11:19:34.361+0000 7f86062c8640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8600135ed0 con 0x7f86000b8240 2024-03-29T11:19:34.350 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T11:19:34.361+0000 7f86062c8640 10 monclient(hunting): _renew_subs 2024-03-29T11:19:34.350 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T11:19:34.361+0000 7f86062c8640 10 monclient(hunting): authenticate will time out at 835.655264s 2024-03-29T11:19:34.350 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T11:19:34.361+0000 7f85ff7fe640 1 --1- >> v1:172.21.15.84:6790/0 conn(0x7f86000b8240 0x7f8600134f60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.84:6790/0 says I am v1:172.21.15.84:36062/0 (socket says 172.21.15.84:36062) 2024-03-29T11:19:34.351 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T11:19:34.361+0000 7f8604865640 1 --1- >> v1:172.21.15.84:6789/0 conn(0x7f8600136410 0x7f86001e2a00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.84:6789/0 says I am v1:172.21.15.84:39064/0 (socket says 172.21.15.84:39064) 2024-03-29T11:19:34.351 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T11:19:34.361+0000 7f85ff7fe640 1 -- 172.21.15.84:0/3943859577 learned_addr learned my addr 172.21.15.84:0/3943859577 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-29T11:19:34.351 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T11:19:34.361+0000 7f85fcff9640 10 client.?.objecter ms_handle_connect 0x7f86000b8240 2024-03-29T11:19:34.351 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T11:19:34.361+0000 7f85fcff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:19:34.351 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T11:19:34.361+0000 7f85fcff9640 10 client.?.objecter ms_handle_connect 0x7f8600136410 2024-03-29T11:19:34.351 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T11:19:34.361+0000 7f85fcff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:19:34.351 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T11:19:34.361+0000 7f85fcff9640 10 client.?.objecter ms_handle_connect 0x7f860009c8c0 2024-03-29T11:19:34.351 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T11:19:34.361+0000 7f85fcff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:19:34.351 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T11:19:34.361+0000 7f85fcff9640 1 -- 172.21.15.84:0/3943859577 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3350744561 0 0) 0x7f8600135ed0 con 0x7f86000b8240 2024-03-29T11:19:34.351 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T11:19:34.361+0000 7f85fcff9640 10 monclient(hunting): _negotiate 2024-03-29T11:19:34.351 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T11:19:34.361+0000 7f85fcff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:34.351 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T11:19:34.361+0000 7f85fcff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:34.351 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T11:19:34.361+0000 7f85fcff9640 10 monclient(hunting): my global_id is 4190 2024-03-29T11:19:34.351 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T11:19:34.361+0000 7f85fcff9640 1 -- 172.21.15.84:0/3943859577 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f85ec004eb0 con 0x7f86000b8240 2024-03-29T11:19:34.351 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T11:19:34.361+0000 7f85fcff9640 1 -- 172.21.15.84:0/3943859577 <== mon.0 v1:172.21.15.84:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 722961422 0 0) 0x7f8600137f80 con 0x7f8600136410 2024-03-29T11:19:34.351 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T11:19:34.361+0000 7f85fcff9640 10 monclient(hunting): _negotiate 2024-03-29T11:19:34.352 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T11:19:34.361+0000 7f85fcff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:34.352 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T11:19:34.361+0000 7f85fcff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:34.352 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T11:19:34.361+0000 7f85fcff9640 10 monclient(hunting): my global_id is 4191 2024-03-29T11:19:34.352 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T11:19:34.361+0000 7f85fcff9640 1 -- 172.21.15.84:0/3943859577 --> v1:172.21.15.84:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8600135ed0 con 0x7f8600136410 2024-03-29T11:19:34.352 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T11:19:34.361+0000 7f85fcff9640 1 -- 172.21.15.84:0/3943859577 <== mon.1 v1:172.21.15.179:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 245594638 0 0) 0x7f8600135c30 con 0x7f860009c8c0 2024-03-29T11:19:34.352 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T11:19:34.361+0000 7f85fcff9640 10 monclient(hunting): _negotiate 2024-03-29T11:19:34.352 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T11:19:34.361+0000 7f85fcff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:34.352 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T11:19:34.361+0000 7f85fcff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:34.352 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T11:19:34.361+0000 7f85fcff9640 10 monclient(hunting): my global_id is 4174 2024-03-29T11:19:34.352 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T11:19:34.361+0000 7f85fcff9640 1 -- 172.21.15.84:0/3943859577 --> v1:172.21.15.179:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8600137f80 con 0x7f860009c8c0 2024-03-29T11:19:34.352 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T11:19:34.361+0000 7f85fcff9640 1 -- 172.21.15.84:0/3943859577 <== mon.2 v1:172.21.15.84:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 385977666 0 0) 0x7f85ec004eb0 con 0x7f86000b8240 2024-03-29T11:19:34.352 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T11:19:34.361+0000 7f85fcff9640 1 -- 172.21.15.84:0/3943859577 --> v1:172.21.15.84:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8600135c30 con 0x7f86000b8240 2024-03-29T11:19:34.352 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T11:19:34.361+0000 7f85fcff9640 1 -- 172.21.15.84:0/3943859577 <== mon.2 v1:172.21.15.84:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4236868694 0 0) 0x7f85f0001b80 con 0x7f86000b8240 2024-03-29T11:19:34.352 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T11:19:34.361+0000 7f85fcff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:34.352 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T11:19:34.361+0000 7f85fcff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:19:34.353 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T11:19:34.361+0000 7f85fcff9640 10 monclient(hunting): dump: 2024-03-29T11:19:34.353 INFO:teuthology.orchestra.run.smithi084.stderr:epoch 1 2024-03-29T11:19:34.353 INFO:teuthology.orchestra.run.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:19:34.353 INFO:teuthology.orchestra.run.smithi084.stderr:last_changed 2024-03-29T11:18:53.864025+0000 2024-03-29T11:19:34.353 INFO:teuthology.orchestra.run.smithi084.stderr:created 2024-03-29T11:18:53.864025+0000 2024-03-29T11:19:34.353 INFO:teuthology.orchestra.run.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:34.353 INFO:teuthology.orchestra.run.smithi084.stderr:election_strategy: 1 2024-03-29T11:19:34.353 INFO:teuthology.orchestra.run.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T11:19:34.353 INFO:teuthology.orchestra.run.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T11:19:34.353 INFO:teuthology.orchestra.run.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T11:19:34.353 INFO:teuthology.orchestra.run.smithi084.stderr: 2024-03-29T11:19:34.353 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T11:19:34.361+0000 7f85fcff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:19:34.353 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T11:19:34.361+0000 7f85fcff9640 1 -- 172.21.15.84:0/3943859577 <== mon.0 v1:172.21.15.84:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4035841344 0 0) 0x7f8600135ed0 con 0x7f8600136410 2024-03-29T11:19:34.353 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T11:19:34.361+0000 7f85fcff9640 1 -- 172.21.15.84:0/3943859577 --> v1:172.21.15.84:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f85ec004eb0 con 0x7f8600136410 2024-03-29T11:19:34.353 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T11:19:34.361+0000 7f85fcff9640 1 -- 172.21.15.84:0/3943859577 <== mon.0 v1:172.21.15.84:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4236868694 0 0) 0x7f85f8001b70 con 0x7f8600136410 2024-03-29T11:19:34.353 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T11:19:34.361+0000 7f85fcff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:34.353 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T11:19:34.361+0000 7f85fcff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:19:34.354 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T11:19:34.361+0000 7f85fcff9640 10 monclient(hunting): dump: 2024-03-29T11:19:34.354 INFO:teuthology.orchestra.run.smithi084.stderr:epoch 1 2024-03-29T11:19:34.354 INFO:teuthology.orchestra.run.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:19:34.354 INFO:teuthology.orchestra.run.smithi084.stderr:last_changed 2024-03-29T11:18:53.864025+0000 2024-03-29T11:19:34.354 INFO:teuthology.orchestra.run.smithi084.stderr:created 2024-03-29T11:18:53.864025+0000 2024-03-29T11:19:34.354 INFO:teuthology.orchestra.run.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:34.354 INFO:teuthology.orchestra.run.smithi084.stderr:election_strategy: 1 2024-03-29T11:19:34.354 INFO:teuthology.orchestra.run.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T11:19:34.354 INFO:teuthology.orchestra.run.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T11:19:34.354 INFO:teuthology.orchestra.run.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T11:19:34.354 INFO:teuthology.orchestra.run.smithi084.stderr: 2024-03-29T11:19:34.354 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T11:19:34.361+0000 7f85fcff9640 1 -- 172.21.15.84:0/3943859577 <== mon.1 v1:172.21.15.179:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 390600924 0 0) 0x7f8600137f80 con 0x7f860009c8c0 2024-03-29T11:19:34.354 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T11:19:34.361+0000 7f85fcff9640 1 -- 172.21.15.84:0/3943859577 --> v1:172.21.15.179:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8600135ed0 con 0x7f860009c8c0 2024-03-29T11:19:34.354 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T11:19:34.361+0000 7f85fcff9640 1 -- 172.21.15.84:0/3943859577 <== mon.1 v1:172.21.15.179:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4236868694 0 0) 0x7f85f4001830 con 0x7f860009c8c0 2024-03-29T11:19:34.355 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T11:19:34.361+0000 7f85fcff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:34.355 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T11:19:34.361+0000 7f85fcff9640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:19:34.355 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T11:19:34.361+0000 7f85fcff9640 10 monclient(hunting): dump: 2024-03-29T11:19:34.355 INFO:teuthology.orchestra.run.smithi084.stderr:epoch 1 2024-03-29T11:19:34.355 INFO:teuthology.orchestra.run.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:19:34.355 INFO:teuthology.orchestra.run.smithi084.stderr:last_changed 2024-03-29T11:18:53.864025+0000 2024-03-29T11:19:34.355 INFO:teuthology.orchestra.run.smithi084.stderr:created 2024-03-29T11:18:53.864025+0000 2024-03-29T11:19:34.355 INFO:teuthology.orchestra.run.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:34.355 INFO:teuthology.orchestra.run.smithi084.stderr:election_strategy: 1 2024-03-29T11:19:34.355 INFO:teuthology.orchestra.run.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T11:19:34.355 INFO:teuthology.orchestra.run.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T11:19:34.355 INFO:teuthology.orchestra.run.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T11:19:34.355 INFO:teuthology.orchestra.run.smithi084.stderr: 2024-03-29T11:19:34.355 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T11:19:34.361+0000 7f85fcff9640 1 -- 172.21.15.84:0/3943859577 <== mon.2 v1:172.21.15.84:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3137385540 0 0) 0x7f8600135c30 con 0x7f86000b8240 2024-03-29T11:19:34.355 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T11:19:34.361+0000 7f85fcff9640 1 -- 172.21.15.84:0/3943859577 >> v1:172.21.15.179:6789/0 conn(0x7f860009c8c0 legacy=0x7f8600143800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:34.355 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T11:19:34.361+0000 7f85fcff9640 1 -- 172.21.15.84:0/3943859577 >> v1:172.21.15.84:6789/0 conn(0x7f8600136410 legacy=0x7f86001e2a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:34.356 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T11:19:34.361+0000 7f85fcff9640 10 monclient: _finish_hunting 0 2024-03-29T11:19:34.356 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T11:19:34.361+0000 7f85fcff9640 1 monclient: found mon.c 2024-03-29T11:19:34.356 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T11:19:34.361+0000 7f85fcff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:19:34.356 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T11:19:34.361+0000 7f85fcff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-29T11:19:34.356 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T11:19:34.361+0000 7f85fcff9640 1 -- 172.21.15.84:0/3943859577 --> v1:172.21.15.84:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8600146260 con 0x7f86000b8240 2024-03-29T11:19:34.356 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T11:19:34.361+0000 7f85fcff9640 10 monclient: _finish_auth 0 2024-03-29T11:19:34.356 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T11:19:34.361+0000 7f85fcff9640 10 monclient: _check_auth_tickets 2024-03-29T11:19:34.356 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T11:19:34.361+0000 7f85fcff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:19:34.356 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T11:19:34.361+0000 7f86062c8640 5 monclient: authenticate success, global_id 4190 2024-03-29T11:19:34.356 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T11:19:34.361+0000 7f86062c8640 10 monclient: _renew_subs 2024-03-29T11:19:34.356 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T11:19:34.361+0000 7f86062c8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-29T11:19:34.356 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T11:19:34.361+0000 7f86062c8640 1 -- 172.21.15.84:0/3943859577 --> v1:172.21.15.84:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8600135670 con 0x7f86000b8240 2024-03-29T11:19:34.356 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T11:19:34.362+0000 7f86062c8640 10 client.4190.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:19:34.357 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T11:19:34.362+0000 7f86062c8640 10 monclient: _renew_subs 2024-03-29T11:19:34.357 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T11:19:34.362+0000 7f86062c8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-29T11:19:34.357 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T11:19:34.362+0000 7f86062c8640 1 -- 172.21.15.84:0/3943859577 --> v1:172.21.15.84:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f86001dab10 con 0x7f86000b8240 2024-03-29T11:19:34.357 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T11:19:34.362+0000 7f86062c8640 1 librados: init done 2024-03-29T11:19:34.357 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T11:19:34.362+0000 7f85fcff9640 1 -- 172.21.15.84:0/3943859577 <== mon.2 v1:172.21.15.84:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f85f0001210 con 0x7f86000b8240 2024-03-29T11:19:34.357 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T11:19:34.362+0000 7f85fcff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:19:34.357 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T11:19:34.362+0000 7f85fcff9640 1 -- 172.21.15.84:0/3943859577 <== mon.2 v1:172.21.15.84:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4236868694 0 0) 0x7f85f0004d90 con 0x7f86000b8240 2024-03-29T11:19:34.357 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T11:19:34.362+0000 7f85fcff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:34.357 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T11:19:34.362+0000 7f85fcff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:19:34.357 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T11:19:34.362+0000 7f85fe7fc640 4 set_mon_vals no callback set 2024-03-29T11:19:34.357 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T11:19:34.362+0000 7f85fcff9640 10 monclient: dump: 2024-03-29T11:19:34.357 INFO:teuthology.orchestra.run.smithi084.stderr:epoch 1 2024-03-29T11:19:34.357 INFO:teuthology.orchestra.run.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:19:34.357 INFO:teuthology.orchestra.run.smithi084.stderr:last_changed 2024-03-29T11:18:53.864025+0000 2024-03-29T11:19:34.357 INFO:teuthology.orchestra.run.smithi084.stderr:created 2024-03-29T11:18:53.864025+0000 2024-03-29T11:19:34.358 INFO:teuthology.orchestra.run.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:34.358 INFO:teuthology.orchestra.run.smithi084.stderr:election_strategy: 1 2024-03-29T11:19:34.358 INFO:teuthology.orchestra.run.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T11:19:34.358 INFO:teuthology.orchestra.run.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T11:19:34.358 INFO:teuthology.orchestra.run.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T11:19:34.358 INFO:teuthology.orchestra.run.smithi084.stderr: 2024-03-29T11:19:34.358 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T11:19:34.362+0000 7f85fcff9640 1 -- 172.21.15.84:0/3943859577 <== mon.2 v1:172.21.15.84:6790/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f85f00052c0 con 0x7f86000b8240 2024-03-29T11:19:34.358 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T11:19:34.362+0000 7f85fcff9640 1 -- 172.21.15.84:0/3943859577 <== mon.2 v1:172.21.15.84:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2808139587 0 0) 0x7f85f0006b20 con 0x7f86000b8240 2024-03-29T11:19:34.358 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T11:19:34.362+0000 7f85fcff9640 10 client.4190.objecter ms_dispatch 0x7f8600000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:19:34.358 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T11:19:34.362+0000 7f85fcff9640 3 client.4190.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:19:34.358 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T11:19:34.362+0000 7f85fcff9640 3 client.4190.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:19:34.358 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T11:19:34.362+0000 7f85fcff9640 20 client.4190.objecter dump_active .. 0 homeless 2024-03-29T11:19:34.358 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T11:19:34.362+0000 7f86062c8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:34.358 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T11:19:34.362+0000 7f86062c8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:34.358 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T11:19:34.362+0000 7f86062c8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-29T11:19:34.359 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T11:19:34.362+0000 7f86062c8640 1 -- 172.21.15.84:0/3943859577 --> v1:172.21.15.84:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f85e0005080 con 0x7f86000b8240 2024-03-29T11:19:34.359 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T11:19:34.365+0000 7f85fcff9640 1 -- 172.21.15.84:0/3943859577 <== mon.2 v1:172.21.15.84:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f85f0006500 con 0x7f86000b8240 2024-03-29T11:19:34.359 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T11:19:34.365+0000 7f85fcff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:34.359 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T11:19:34.365+0000 7f85fcff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:19:34.368 INFO:tasks.ceph.mgr.y.smithi084.stderr:2024-03-29T11:19:34.391+0000 7f45957a8040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-29T11:19:34.403 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T11:19:34.426+0000 7f86062c8640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:19:34.403 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T11:19:34.426+0000 7f86062c8640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:19:34.403 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T11:19:34.426+0000 7f86062c8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-29T11:19:34.403 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T11:19:34.426+0000 7f86062c8640 1 -- 172.21.15.84:0/3943859577 --> v1:172.21.15.84:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f85e0005330 con 0x7f86000b8240 2024-03-29T11:19:34.403 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T11:19:34.427+0000 7f85fcff9640 1 -- 172.21.15.84:0/3943859577 <== mon.2 v1:172.21.15.84:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4209477549) 0x7f85f00068b0 con 0x7f86000b8240 2024-03-29T11:19:34.404 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T11:19:34.427+0000 7f85fcff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:19:34.404 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T11:19:34.427+0000 7f85fcff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:19:34.404 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-03-29T11:19:34.404 INFO:teuthology.orchestra.run.smithi084.stdout:{"epoch":10,"fsid":"2c3e002f-9eb0-4860-853b-4ef1ef562213","created":"2024-03-29T11:19:32.029169+0000","modified":"2024-03-29T11:19:34.200299+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:34.200299+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"5c39e276-2034-4716-bbc0-57a634eefa92","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"cd580e99-109f-4bf1-88c0-a5987bc8b9ed","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"11e60a5a-051d-453f-a190-66387cb35592","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"2c329fbe-bbc4-4030-bf87-d711cad04679","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f4213d4a-bd36-4433-91ae-d7e2b02481d3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"f0cf44ea-6055-48af-805b-aed858102ff9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"d895eaa0-cc0b-4ee4-9b5a-2dfcd89a60b7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"9bc5a7bc-8f52-4db9-96d6-de5e68d52eaa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:19:34.405 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T11:19:34.427+0000 7f86062c8640 10 librados: watch_flush enter 2024-03-29T11:19:34.405 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T11:19:34.427+0000 7f86062c8640 10 librados: watch_flush exit 2024-03-29T11:19:34.405 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T11:19:34.427+0000 7f86062c8640 20 client.4190.objecter shutdown clearing up homeless session... 2024-03-29T11:19:34.405 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T11:19:34.427+0000 7f86062c8640 10 client.4190.objecter successfully canceled tick 2024-03-29T11:19:34.406 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T11:19:34.427+0000 7f86062c8640 10 monclient: shutdown 2024-03-29T11:19:34.406 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T11:19:34.427+0000 7f86062c8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:19:34.406 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T11:19:34.427+0000 7f86062c8640 1 -- 172.21.15.84:0/3943859577 >> v1:172.21.15.84:6790/0 conn(0x7f86000b8240 legacy=0x7f8600134f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:34.406 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T11:19:34.427+0000 7f86062c8640 1 -- 172.21.15.84:0/3943859577 shutdown_connections 2024-03-29T11:19:34.406 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T11:19:34.427+0000 7f86062c8640 1 -- 172.21.15.84:0/3943859577 >> 172.21.15.84:0/3943859577 conn(0x7f8600066b00 msgr2=0x7f8600069bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:19:34.406 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T11:19:34.428+0000 7f86062c8640 1 -- 172.21.15.84:0/3943859577 shutdown_connections 2024-03-29T11:19:34.406 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T11:19:34.428+0000 7f86062c8640 1 -- 172.21.15.84:0/3943859577 wait complete. 2024-03-29T11:19:34.406 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T11:19:34.428+0000 7f86062c8640 1 librados: shutdown 2024-03-29T11:19:34.414 INFO:tasks.ceph.ceph_manager.ceph:[] 2024-03-29T11:19:34.415 INFO:tasks.ceph:Waiting for OSDs to come up 2024-03-29T11:19:34.417 INFO:tasks.ceph.mgr.x.smithi179.stderr:2024-03-29T11:19:34.440+0000 7f8706237040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-29T11:19:34.462 INFO:tasks.ceph.mgr.x.smithi179.stderr:2024-03-29T11:19:34.485+0000 7f8706237040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-29T11:19:34.484 INFO:tasks.ceph.mgr.y.smithi084.stderr:2024-03-29T11:19:34.507+0000 7f45957a8040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-29T11:19:34.575 INFO:tasks.ceph.mgr.x.smithi179.stderr:2024-03-29T11:19:34.598+0000 7f8706237040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-29T11:19:34.579 INFO:tasks.ceph.mgr.y.smithi084.stderr:2024-03-29T11:19:34.603+0000 7f45957a8040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-29T11:19:34.670 INFO:tasks.ceph.mgr.x.smithi179.stderr:2024-03-29T11:19:34.693+0000 7f8706237040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-29T11:19:34.732 INFO:tasks.ceph.mgr.y.smithi084.stderr:2024-03-29T11:19:34.756+0000 7f45957a8040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-29T11:19:34.774 INFO:tasks.ceph.mgr.y.smithi084.stderr:2024-03-29T11:19:34.797+0000 7f45957a8040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-29T11:19:34.796 INFO:tasks.ceph.osd.0.smithi084.stderr:2024-03-29T11:19:34.819+0000 7f36b7111640 -1 Falling back to public interface 2024-03-29T11:19:34.799 INFO:tasks.ceph.osd.4.smithi179.stderr:2024-03-29T11:19:34.823+0000 7fc7d0d11640 -1 Falling back to public interface 2024-03-29T11:19:34.803 INFO:tasks.ceph.osd.7.smithi179.stderr:2024-03-29T11:19:34.826+0000 7ffa32d21640 -1 Falling back to public interface 2024-03-29T11:19:34.810 INFO:tasks.ceph.osd.3.smithi084.stderr:2024-03-29T11:19:34.833+0000 7f56cfcf5640 -1 Falling back to public interface 2024-03-29T11:19:34.814 INFO:tasks.ceph.osd.1.smithi084.stderr:2024-03-29T11:19:34.837+0000 7fe314538640 -1 Falling back to public interface 2024-03-29T11:19:34.818 INFO:tasks.ceph.osd.2.smithi084.stderr:2024-03-29T11:19:34.842+0000 7eff51a24640 -1 Falling back to public interface 2024-03-29T11:19:34.829 INFO:tasks.ceph.mgr.x.smithi179.stderr:2024-03-29T11:19:34.852+0000 7f8706237040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-29T11:19:34.831 INFO:tasks.ceph.osd.5.smithi179.stderr:2024-03-29T11:19:34.854+0000 7fa54308b640 -1 Falling back to public interface 2024-03-29T11:19:34.836 INFO:tasks.ceph.osd.6.smithi179.stderr:2024-03-29T11:19:34.859+0000 7f5149822640 -1 Falling back to public interface 2024-03-29T11:19:34.872 INFO:tasks.ceph.mgr.x.smithi179.stderr:2024-03-29T11:19:34.896+0000 7f8706237040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-29T11:19:34.934 INFO:tasks.ceph.mgr.y.smithi084.stderr:2024-03-29T11:19:34.957+0000 7f45957a8040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-29T11:19:35.027 INFO:tasks.ceph.mgr.x.smithi179.stderr:2024-03-29T11:19:35.050+0000 7f8706237040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-29T11:19:35.222 DEBUG:teuthology.orchestra.run.smithi084:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:19:35.311 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:35.334+0000 7f380fc4a640 10 monclient: get_monmap_and_config 2024-03-29T11:19:35.311 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:35.334+0000 7f380fc4a640 10 monclient: build_initial_monmap 2024-03-29T11:19:35.311 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:35.334+0000 7f380fc4a640 10 monclient: monmap: 2024-03-29T11:19:35.311 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-29T11:19:35.311 INFO:teuthology.misc.health.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:19:35.311 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T11:19:35.335253+0000 2024-03-29T11:19:35.311 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T11:19:35.335253+0000 2024-03-29T11:19:35.311 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-29T11:19:35.311 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:19:35.311 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-29T11:19:35.311 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.noname-c 2024-03-29T11:19:35.311 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-b 2024-03-29T11:19:35.312 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:19:35.319 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:35.334+0000 7f380fc4a640 1 Processor -- start 2024-03-29T11:19:35.319 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:35.335+0000 7f380fc4a640 1 -- start start 2024-03-29T11:19:35.319 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:35.335+0000 7f380fc4a640 10 monclient: init 2024-03-29T11:19:35.319 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:35.335+0000 7f380fc4a640 10 monclient: _reopen_session rank -1 2024-03-29T11:19:35.319 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:35.335+0000 7f380fc4a640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:19:35.319 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:35.335+0000 7f380fc4a640 10 monclient(hunting): picked mon.noname-c con 0x7f38081422b0 addr v1:172.21.15.179:6789/0 2024-03-29T11:19:35.319 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:35.335+0000 7f380fc4a640 10 monclient(hunting): picked mon.noname-a con 0x7f3808142690 addr v1:172.21.15.84:6789/0 2024-03-29T11:19:35.319 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:35.335+0000 7f380fc4a640 10 monclient(hunting): picked mon.noname-b con 0x7f38080b17d0 addr v1:172.21.15.84:6790/0 2024-03-29T11:19:35.319 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:35.335+0000 7f380fc4a640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3808142c80 con 0x7f3808142690 2024-03-29T11:19:35.320 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:35.335+0000 7f380fc4a640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f38080aae40 con 0x7f38081422b0 2024-03-29T11:19:35.320 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:35.335+0000 7f380fc4a640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f380813fbd0 con 0x7f38080b17d0 2024-03-29T11:19:35.320 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:35.335+0000 7f380fc4a640 10 monclient(hunting): _renew_subs 2024-03-29T11:19:35.320 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:35.335+0000 7f380fc4a640 10 monclient(hunting): authenticate will time out at 836.629982s 2024-03-29T11:19:35.320 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:35.335+0000 7f380e1e7640 1 --1- >> v1:172.21.15.84:6790/0 conn(0x7f38080b17d0 0x7f38080b2ca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.84:6790/0 says I am v1:172.21.15.84:36076/0 (socket says 172.21.15.84:36076) 2024-03-29T11:19:35.320 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:35.335+0000 7f380d1e5640 1 --1- >> v1:172.21.15.84:6789/0 conn(0x7f3808142690 0x7f38080b2590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.84:6789/0 says I am v1:172.21.15.84:39066/0 (socket says 172.21.15.84:39066) 2024-03-29T11:19:35.320 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:35.335+0000 7f380e1e7640 1 -- 172.21.15.84:0/3994146466 learned_addr learned my addr 172.21.15.84:0/3994146466 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-29T11:19:35.320 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:35.336+0000 7f380c9e4640 1 -- 172.21.15.84:0/3994146466 <== mon.0 v1:172.21.15.84:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1794166985 0 0) 0x7f3808142c80 con 0x7f3808142690 2024-03-29T11:19:35.320 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:35.336+0000 7f380c9e4640 10 monclient(hunting): _negotiate 2024-03-29T11:19:35.320 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:35.336+0000 7f380c9e4640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:35.320 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:35.336+0000 7f380c9e4640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:35.320 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:35.336+0000 7f380c9e4640 10 monclient(hunting): my global_id is 4194 2024-03-29T11:19:35.320 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:35.336+0000 7f380c9e4640 1 -- 172.21.15.84:0/3994146466 --> v1:172.21.15.84:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f37f4003520 con 0x7f3808142690 2024-03-29T11:19:35.320 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:35.336+0000 7f380c9e4640 1 -- 172.21.15.84:0/3994146466 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1206960953 0 0) 0x7f380813fbd0 con 0x7f38080b17d0 2024-03-29T11:19:35.320 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:35.336+0000 7f380c9e4640 10 monclient(hunting): _negotiate 2024-03-29T11:19:35.321 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:35.336+0000 7f380c9e4640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:35.321 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:35.336+0000 7f380c9e4640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:35.321 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:35.336+0000 7f380c9e4640 10 monclient(hunting): my global_id is 4193 2024-03-29T11:19:35.321 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:35.336+0000 7f380c9e4640 1 -- 172.21.15.84:0/3994146466 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3808142c80 con 0x7f38080b17d0 2024-03-29T11:19:35.321 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:35.336+0000 7f380c9e4640 1 -- 172.21.15.84:0/3994146466 <== mon.1 v1:172.21.15.179:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3658887543 0 0) 0x7f38080aae40 con 0x7f38081422b0 2024-03-29T11:19:35.321 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:35.336+0000 7f380c9e4640 10 monclient(hunting): _negotiate 2024-03-29T11:19:35.321 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:35.336+0000 7f380c9e4640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:35.321 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:35.336+0000 7f380c9e4640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:35.321 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:35.336+0000 7f380c9e4640 10 monclient(hunting): my global_id is 4177 2024-03-29T11:19:35.321 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:35.336+0000 7f380c9e4640 1 -- 172.21.15.84:0/3994146466 --> v1:172.21.15.179:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f380813fbd0 con 0x7f38081422b0 2024-03-29T11:19:35.321 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:35.336+0000 7f380c9e4640 1 -- 172.21.15.84:0/3994146466 <== mon.0 v1:172.21.15.84:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1173505806 0 0) 0x7f37f4003520 con 0x7f3808142690 2024-03-29T11:19:35.321 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:35.336+0000 7f380c9e4640 1 -- 172.21.15.84:0/3994146466 >> v1:172.21.15.84:6790/0 conn(0x7f38080b17d0 legacy=0x7f38080b2ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:35.321 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:35.336+0000 7f380c9e4640 1 -- 172.21.15.84:0/3994146466 >> v1:172.21.15.179:6789/0 conn(0x7f38081422b0 legacy=0x7f380813f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:35.321 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:35.336+0000 7f380c9e4640 10 monclient: _finish_hunting 0 2024-03-29T11:19:35.321 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:35.336+0000 7f380c9e4640 1 monclient: found mon.noname-a 2024-03-29T11:19:35.321 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:35.336+0000 7f380c9e4640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:19:35.321 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:35.336+0000 7f380c9e4640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.84:6789/0 2024-03-29T11:19:35.322 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:35.336+0000 7f380c9e4640 1 -- 172.21.15.84:0/3994146466 --> v1:172.21.15.84:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3808151940 con 0x7f3808142690 2024-03-29T11:19:35.322 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:35.336+0000 7f380c9e4640 10 monclient: _finish_auth 0 2024-03-29T11:19:35.322 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:35.336+0000 7f380c9e4640 10 monclient: _check_auth_tickets 2024-03-29T11:19:35.322 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:35.336+0000 7f380c9e4640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:19:35.322 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:35.336+0000 7f380c9e4640 1 -- 172.21.15.84:0/3994146466 <== mon.0 v1:172.21.15.84:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4236868694 0 0) 0x7f37f8001690 con 0x7f3808142690 2024-03-29T11:19:35.322 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:35.336+0000 7f380fc4a640 5 monclient: authenticate success, global_id 4194 2024-03-29T11:19:35.322 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:35.336+0000 7f380fc4a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:19:35.322 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:35.336+0000 7f380c9e4640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:35.322 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:35.336+0000 7f380c9e4640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:19:35.322 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:35.336+0000 7f380c9e4640 10 monclient: dump: 2024-03-29T11:19:35.322 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T11:19:35.322 INFO:teuthology.misc.health.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:19:35.322 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T11:18:53.864025+0000 2024-03-29T11:19:35.322 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T11:18:53.864025+0000 2024-03-29T11:19:35.323 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:35.323 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:19:35.323 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T11:19:35.323 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T11:19:35.323 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T11:19:35.323 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:19:35.323 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:35.336+0000 7f380fc4a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:19:35.323 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:35.336+0000 7f380c9e4640 1 -- 172.21.15.84:0/3994146466 <== mon.0 v1:172.21.15.84:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f37f80017f0 con 0x7f3808142690 2024-03-29T11:19:35.323 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:35.336+0000 7f380c9e4640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:19:35.323 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:35.336+0000 7f380c9e4640 1 -- 172.21.15.84:0/3994146466 <== mon.0 v1:172.21.15.84:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4236868694 0 0) 0x7f37f8001bc0 con 0x7f3808142690 2024-03-29T11:19:35.323 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:35.336+0000 7f380c9e4640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:35.323 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:35.336+0000 7f380c9e4640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:19:35.323 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:35.336+0000 7f380c9e4640 10 monclient: dump: 2024-03-29T11:19:35.323 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T11:19:35.323 INFO:teuthology.misc.health.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:19:35.323 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T11:18:53.864025+0000 2024-03-29T11:19:35.324 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T11:18:53.864025+0000 2024-03-29T11:19:35.324 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:35.324 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:19:35.324 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T11:19:35.324 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T11:19:35.324 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T11:19:35.324 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:19:35.324 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:35.336+0000 7f380fc4a640 10 monclient: get_monmap_and_config success 2024-03-29T11:19:35.324 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:35.336+0000 7f380fc4a640 4 set_mon_vals no callback set 2024-03-29T11:19:35.324 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:35.337+0000 7f380fc4a640 10 monclient: shutdown 2024-03-29T11:19:35.324 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:35.337+0000 7f380fc4a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:19:35.324 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:35.337+0000 7f380fc4a640 1 -- 172.21.15.84:0/3994146466 >> v1:172.21.15.84:6789/0 conn(0x7f3808142690 legacy=0x7f38080b2590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:35.324 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:35.337+0000 7f380fc4a640 1 -- 172.21.15.84:0/3994146466 shutdown_connections 2024-03-29T11:19:35.324 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:35.337+0000 7f380fc4a640 1 -- 172.21.15.84:0/3994146466 >> 172.21.15.84:0/3994146466 conn(0x7f3808066a00 msgr2=0x7f3808066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:19:35.324 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:35.337+0000 7f380fc4a640 1 -- 172.21.15.84:0/3994146466 shutdown_connections 2024-03-29T11:19:35.324 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:35.337+0000 7f380fc4a640 1 -- 172.21.15.84:0/3994146466 wait complete. 2024-03-29T11:19:35.325 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:35.337+0000 7f380fc4a640 10 monclient: build_initial_monmap 2024-03-29T11:19:35.325 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:35.337+0000 7f380fc4a640 10 monclient: monmap: 2024-03-29T11:19:35.325 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-29T11:19:35.325 INFO:teuthology.misc.health.smithi084.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:19:35.325 INFO:teuthology.misc.health.smithi084.stderr:last_changed 0.000000 2024-03-29T11:19:35.325 INFO:teuthology.misc.health.smithi084.stderr:created 0.000000 2024-03-29T11:19:35.325 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-29T11:19:35.325 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:19:35.325 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-29T11:19:35.325 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.noname-b 2024-03-29T11:19:35.325 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-c 2024-03-29T11:19:35.325 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:19:35.325 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:35.338+0000 7f380fc4a640 1 librados: starting msgr at 2024-03-29T11:19:35.325 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:35.338+0000 7f380fc4a640 1 librados: starting objecter 2024-03-29T11:19:35.325 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:35.338+0000 7f380fc4a640 1 Processor -- start 2024-03-29T11:19:35.325 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:35.338+0000 7f380fc4a640 1 -- start start 2024-03-29T11:19:35.325 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:35.338+0000 7f380fc4a640 1 librados: setting wanted keys 2024-03-29T11:19:35.326 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:35.338+0000 7f380fc4a640 1 librados: calling monclient init 2024-03-29T11:19:35.326 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:35.338+0000 7f380fc4a640 10 monclient: init 2024-03-29T11:19:35.326 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:35.338+0000 7f380fc4a640 10 monclient: _reopen_session rank -1 2024-03-29T11:19:35.326 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:35.338+0000 7f380fc4a640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:19:35.326 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:35.338+0000 7f380fc4a640 10 monclient(hunting): picked mon.noname-a con 0x7f38080b17d0 addr v1:172.21.15.84:6789/0 2024-03-29T11:19:35.326 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:35.338+0000 7f380fc4a640 10 monclient(hunting): picked mon.noname-c con 0x7f38081422b0 addr v1:172.21.15.84:6790/0 2024-03-29T11:19:35.326 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:35.338+0000 7f380fc4a640 10 monclient(hunting): picked mon.noname-b con 0x7f3808142690 addr v1:172.21.15.179:6789/0 2024-03-29T11:19:35.326 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:35.338+0000 7f380fc4a640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f380812da20 con 0x7f38080b17d0 2024-03-29T11:19:35.326 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:35.338+0000 7f380fc4a640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f38080acac0 con 0x7f3808142690 2024-03-29T11:19:35.326 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:35.338+0000 7f380fc4a640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f38080acda0 con 0x7f38081422b0 2024-03-29T11:19:35.326 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:35.338+0000 7f380fc4a640 10 monclient(hunting): _renew_subs 2024-03-29T11:19:35.326 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:35.338+0000 7f380fc4a640 10 monclient(hunting): authenticate will time out at 836.632964s 2024-03-29T11:19:35.326 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:35.338+0000 7f380d9e6640 1 --1- >> v1:172.21.15.84:6789/0 conn(0x7f38080b17d0 0x7f380814f290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.84:6789/0 says I am v1:172.21.15.84:39074/0 (socket says 172.21.15.84:39074) 2024-03-29T11:19:35.326 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:35.338+0000 7f380d9e6640 1 -- 172.21.15.84:0/1723251257 learned_addr learned my addr 172.21.15.84:0/1723251257 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-29T11:19:35.326 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:35.338+0000 7f37fe7fc640 10 client.?.objecter ms_handle_connect 0x7f38080b17d0 2024-03-29T11:19:35.327 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:35.339+0000 7f37fe7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:19:35.327 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:35.339+0000 7f37fe7fc640 10 client.?.objecter ms_handle_connect 0x7f38081422b0 2024-03-29T11:19:35.327 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:35.339+0000 7f37fe7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:19:35.327 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:35.339+0000 7f37fe7fc640 1 -- 172.21.15.84:0/1723251257 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1875536095 0 0) 0x7f38080acda0 con 0x7f38081422b0 2024-03-29T11:19:35.327 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:35.339+0000 7f37fe7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:19:35.327 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:35.339+0000 7f37fe7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:35.327 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:35.339+0000 7f37fe7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:35.327 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:35.339+0000 7f37fe7fc640 10 monclient(hunting): my global_id is 4196 2024-03-29T11:19:35.327 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:35.339+0000 7f37fe7fc640 1 -- 172.21.15.84:0/1723251257 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f37ec004eb0 con 0x7f38081422b0 2024-03-29T11:19:35.327 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:35.339+0000 7f37fe7fc640 10 client.?.objecter ms_handle_connect 0x7f3808142690 2024-03-29T11:19:35.327 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:35.339+0000 7f37fe7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:19:35.327 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:35.339+0000 7f37fe7fc640 1 -- 172.21.15.84:0/1723251257 <== mon.0 v1:172.21.15.84:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4278671556 0 0) 0x7f380812da20 con 0x7f38080b17d0 2024-03-29T11:19:35.327 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:35.339+0000 7f37fe7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:19:35.327 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:35.339+0000 7f37fe7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:35.327 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:35.339+0000 7f37fe7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:35.327 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:35.339+0000 7f37fe7fc640 10 monclient(hunting): my global_id is 4197 2024-03-29T11:19:35.328 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:35.339+0000 7f37fe7fc640 1 -- 172.21.15.84:0/1723251257 --> v1:172.21.15.84:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f38080acda0 con 0x7f38080b17d0 2024-03-29T11:19:35.328 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:35.339+0000 7f37fe7fc640 1 -- 172.21.15.84:0/1723251257 <== mon.1 v1:172.21.15.179:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2706818054 0 0) 0x7f38080acac0 con 0x7f3808142690 2024-03-29T11:19:35.328 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:35.339+0000 7f37fe7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:19:35.328 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:35.339+0000 7f37fe7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:35.328 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:35.339+0000 7f37fe7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:35.328 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:35.339+0000 7f37fe7fc640 10 monclient(hunting): my global_id is 4180 2024-03-29T11:19:35.328 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:35.339+0000 7f37fe7fc640 1 -- 172.21.15.84:0/1723251257 --> v1:172.21.15.179:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f380812da20 con 0x7f3808142690 2024-03-29T11:19:35.328 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:35.339+0000 7f37fe7fc640 1 -- 172.21.15.84:0/1723251257 <== mon.2 v1:172.21.15.84:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2826801285 0 0) 0x7f37ec004eb0 con 0x7f38081422b0 2024-03-29T11:19:35.328 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:35.339+0000 7f37fe7fc640 1 -- 172.21.15.84:0/1723251257 --> v1:172.21.15.84:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f38080acac0 con 0x7f38081422b0 2024-03-29T11:19:35.328 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:35.339+0000 7f37fe7fc640 1 -- 172.21.15.84:0/1723251257 <== mon.2 v1:172.21.15.84:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4236868694 0 0) 0x7f37f8001bc0 con 0x7f38081422b0 2024-03-29T11:19:35.328 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:35.339+0000 7f37fe7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:35.328 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:35.339+0000 7f37fe7fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:19:35.328 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:35.339+0000 7f37fe7fc640 10 monclient(hunting): dump: 2024-03-29T11:19:35.328 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T11:19:35.328 INFO:teuthology.misc.health.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:19:35.328 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T11:18:53.864025+0000 2024-03-29T11:19:35.329 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T11:18:53.864025+0000 2024-03-29T11:19:35.329 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:35.329 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:19:35.329 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T11:19:35.329 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T11:19:35.329 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T11:19:35.329 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:19:35.329 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:35.339+0000 7f37fe7fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:19:35.329 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:35.339+0000 7f37fe7fc640 1 -- 172.21.15.84:0/1723251257 <== mon.0 v1:172.21.15.84:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 387319415 0 0) 0x7f38080acda0 con 0x7f38080b17d0 2024-03-29T11:19:35.329 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:35.339+0000 7f37fe7fc640 1 -- 172.21.15.84:0/1723251257 --> v1:172.21.15.84:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f37ec004eb0 con 0x7f38080b17d0 2024-03-29T11:19:35.329 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:35.339+0000 7f37fe7fc640 1 -- 172.21.15.84:0/1723251257 <== mon.0 v1:172.21.15.84:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4236868694 0 0) 0x7f3804001910 con 0x7f38080b17d0 2024-03-29T11:19:35.329 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:35.339+0000 7f37fe7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:35.329 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:35.339+0000 7f37fe7fc640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:19:35.329 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:35.339+0000 7f37fe7fc640 10 monclient(hunting): dump: 2024-03-29T11:19:35.329 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T11:19:35.330 INFO:teuthology.misc.health.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:19:35.330 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T11:18:53.864025+0000 2024-03-29T11:19:35.330 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T11:18:53.864025+0000 2024-03-29T11:19:35.330 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:35.330 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:19:35.330 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T11:19:35.330 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T11:19:35.330 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T11:19:35.330 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:19:35.330 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:35.339+0000 7f37fe7fc640 1 -- 172.21.15.84:0/1723251257 <== mon.1 v1:172.21.15.179:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3592525332 0 0) 0x7f380812da20 con 0x7f3808142690 2024-03-29T11:19:35.330 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:35.339+0000 7f37fe7fc640 1 -- 172.21.15.84:0/1723251257 --> v1:172.21.15.179:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f38080acda0 con 0x7f3808142690 2024-03-29T11:19:35.330 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:35.339+0000 7f37fe7fc640 1 -- 172.21.15.84:0/1723251257 <== mon.1 v1:172.21.15.179:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4236868694 0 0) 0x7f3800001a30 con 0x7f3808142690 2024-03-29T11:19:35.330 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:35.339+0000 7f37fe7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:35.330 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:35.339+0000 7f37fe7fc640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:19:35.331 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:35.339+0000 7f37fe7fc640 10 monclient(hunting): dump: 2024-03-29T11:19:35.331 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T11:19:35.331 INFO:teuthology.misc.health.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:19:35.331 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T11:18:53.864025+0000 2024-03-29T11:19:35.331 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T11:18:53.864025+0000 2024-03-29T11:19:35.331 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:35.331 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:19:35.331 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T11:19:35.331 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T11:19:35.331 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T11:19:35.331 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:19:35.331 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:35.339+0000 7f37fe7fc640 1 -- 172.21.15.84:0/1723251257 <== mon.2 v1:172.21.15.84:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2917296053 0 0) 0x7f38080acac0 con 0x7f38081422b0 2024-03-29T11:19:35.331 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:35.339+0000 7f37fe7fc640 1 -- 172.21.15.84:0/1723251257 >> v1:172.21.15.179:6789/0 conn(0x7f3808142690 legacy=0x7f380812d310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:35.331 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:35.339+0000 7f37fe7fc640 1 -- 172.21.15.84:0/1723251257 >> v1:172.21.15.84:6789/0 conn(0x7f38080b17d0 legacy=0x7f380814f290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:35.331 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:35.339+0000 7f37fe7fc640 10 monclient: _finish_hunting 0 2024-03-29T11:19:35.331 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:35.339+0000 7f37fe7fc640 1 monclient: found mon.c 2024-03-29T11:19:35.331 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:35.339+0000 7f37fe7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:19:35.332 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:35.339+0000 7f37fe7fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-29T11:19:35.332 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:35.339+0000 7f37fe7fc640 1 -- 172.21.15.84:0/1723251257 --> v1:172.21.15.84:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38081509a0 con 0x7f38081422b0 2024-03-29T11:19:35.332 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:35.339+0000 7f37fe7fc640 10 monclient: _finish_auth 0 2024-03-29T11:19:35.332 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:35.339+0000 7f37fe7fc640 10 monclient: _check_auth_tickets 2024-03-29T11:19:35.332 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:35.339+0000 7f37fe7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:19:35.332 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:35.339+0000 7f380fc4a640 5 monclient: authenticate success, global_id 4196 2024-03-29T11:19:35.332 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:35.339+0000 7f380fc4a640 10 monclient: _renew_subs 2024-03-29T11:19:35.332 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:35.339+0000 7f380fc4a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-29T11:19:35.332 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:35.339+0000 7f380fc4a640 1 -- 172.21.15.84:0/1723251257 --> v1:172.21.15.84:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3808150670 con 0x7f38081422b0 2024-03-29T11:19:35.332 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:35.339+0000 7f380fc4a640 10 client.4196.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:19:35.332 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:35.339+0000 7f380fc4a640 10 monclient: _renew_subs 2024-03-29T11:19:35.332 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:35.339+0000 7f380fc4a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-29T11:19:35.332 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:35.339+0000 7f380fc4a640 1 -- 172.21.15.84:0/1723251257 --> v1:172.21.15.84:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f3808151310 con 0x7f38081422b0 2024-03-29T11:19:35.332 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:35.339+0000 7f380fc4a640 1 librados: init done 2024-03-29T11:19:35.332 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:35.339+0000 7f37fe7fc640 1 -- 172.21.15.84:0/1723251257 <== mon.2 v1:172.21.15.84:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f37f8004a90 con 0x7f38081422b0 2024-03-29T11:19:35.332 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:35.339+0000 7f37fe7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:19:35.333 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:35.339+0000 7f37fe7fc640 1 -- 172.21.15.84:0/1723251257 <== mon.2 v1:172.21.15.84:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4236868694 0 0) 0x7f37f8004da0 con 0x7f38081422b0 2024-03-29T11:19:35.333 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:35.339+0000 7f37fe7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:35.333 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:35.339+0000 7f37fe7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:19:35.333 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:35.339+0000 7f37fffff640 4 set_mon_vals no callback set 2024-03-29T11:19:35.333 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:35.339+0000 7f37fe7fc640 10 monclient: dump: 2024-03-29T11:19:35.333 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T11:19:35.333 INFO:teuthology.misc.health.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:19:35.333 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T11:18:53.864025+0000 2024-03-29T11:19:35.333 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T11:18:53.864025+0000 2024-03-29T11:19:35.333 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:35.333 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:19:35.333 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T11:19:35.333 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T11:19:35.333 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T11:19:35.333 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:19:35.333 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:35.339+0000 7f37fe7fc640 1 -- 172.21.15.84:0/1723251257 <== mon.2 v1:172.21.15.84:6790/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f37f8005290 con 0x7f38081422b0 2024-03-29T11:19:35.333 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:35.339+0000 7f37fe7fc640 1 -- 172.21.15.84:0/1723251257 <== mon.2 v1:172.21.15.84:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2808139587 0 0) 0x7f37f8005c10 con 0x7f38081422b0 2024-03-29T11:19:35.334 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:35.339+0000 7f37fe7fc640 10 client.4196.objecter ms_dispatch 0x7f3808000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:19:35.334 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:35.339+0000 7f37fe7fc640 3 client.4196.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:19:35.334 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:35.339+0000 7f37fe7fc640 3 client.4196.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:19:35.334 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:35.339+0000 7f37fe7fc640 20 client.4196.objecter dump_active .. 0 homeless 2024-03-29T11:19:35.334 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:35.340+0000 7f380fc4a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:35.334 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:35.340+0000 7f380fc4a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:35.334 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:35.340+0000 7f380fc4a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-29T11:19:35.334 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:35.340+0000 7f380fc4a640 1 -- 172.21.15.84:0/1723251257 --> v1:172.21.15.84:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f37f0005080 con 0x7f38081422b0 2024-03-29T11:19:35.334 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:35.342+0000 7f37fe7fc640 1 -- 172.21.15.84:0/1723251257 <== mon.2 v1:172.21.15.84:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f37f8005540 con 0x7f38081422b0 2024-03-29T11:19:35.334 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:35.342+0000 7f37fe7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:35.334 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:35.342+0000 7f37fe7fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:19:35.380 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:35.403+0000 7f380fc4a640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:19:35.380 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:35.403+0000 7f380fc4a640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:19:35.380 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:35.403+0000 7f380fc4a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-29T11:19:35.381 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:35.403+0000 7f380fc4a640 1 -- 172.21.15.84:0/1723251257 --> v1:172.21.15.84:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f37f0005330 con 0x7f38081422b0 2024-03-29T11:19:35.381 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:35.404+0000 7f37fe7fc640 1 -- 172.21.15.84:0/1723251257 <== mon.2 v1:172.21.15.84:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4209477549) 0x7f37f80057a0 con 0x7f38081422b0 2024-03-29T11:19:35.381 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:35.404+0000 7f37fe7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:19:35.381 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:35.404+0000 7f37fe7fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:19:35.382 INFO:teuthology.misc.health.smithi084.stdout: 2024-03-29T11:19:35.382 INFO:teuthology.misc.health.smithi084.stdout:{"epoch":10,"fsid":"2c3e002f-9eb0-4860-853b-4ef1ef562213","created":"2024-03-29T11:19:32.029169+0000","modified":"2024-03-29T11:19:34.200299+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:34.200299+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"5c39e276-2034-4716-bbc0-57a634eefa92","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"cd580e99-109f-4bf1-88c0-a5987bc8b9ed","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"11e60a5a-051d-453f-a190-66387cb35592","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"2c329fbe-bbc4-4030-bf87-d711cad04679","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f4213d4a-bd36-4433-91ae-d7e2b02481d3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"f0cf44ea-6055-48af-805b-aed858102ff9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"d895eaa0-cc0b-4ee4-9b5a-2dfcd89a60b7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"9bc5a7bc-8f52-4db9-96d6-de5e68d52eaa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:19:35.383 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:35.405+0000 7f380fc4a640 10 librados: watch_flush enter 2024-03-29T11:19:35.383 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:35.405+0000 7f380fc4a640 10 librados: watch_flush exit 2024-03-29T11:19:35.383 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:35.405+0000 7f380fc4a640 20 client.4196.objecter shutdown clearing up homeless session... 2024-03-29T11:19:35.383 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:35.405+0000 7f380fc4a640 10 client.4196.objecter successfully canceled tick 2024-03-29T11:19:35.383 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:35.405+0000 7f380fc4a640 10 monclient: shutdown 2024-03-29T11:19:35.383 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:35.405+0000 7f380fc4a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:19:35.383 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:35.405+0000 7f380fc4a640 1 -- 172.21.15.84:0/1723251257 >> v1:172.21.15.84:6790/0 conn(0x7f38081422b0 legacy=0x7f380812cc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:35.383 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:35.405+0000 7f380fc4a640 1 -- 172.21.15.84:0/1723251257 shutdown_connections 2024-03-29T11:19:35.383 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:35.405+0000 7f380fc4a640 1 -- 172.21.15.84:0/1723251257 >> 172.21.15.84:0/1723251257 conn(0x7f3808066a00 msgr2=0x7f380813c310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:19:35.383 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:35.405+0000 7f380fc4a640 1 -- 172.21.15.84:0/1723251257 shutdown_connections 2024-03-29T11:19:35.383 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:35.405+0000 7f380fc4a640 1 -- 172.21.15.84:0/1723251257 wait complete. 2024-03-29T11:19:35.384 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:35.405+0000 7f380fc4a640 1 librados: shutdown 2024-03-29T11:19:35.390 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:19:35.850 INFO:tasks.ceph.mgr.x.smithi179.stderr:2024-03-29T11:19:35.874+0000 7f8706237040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-29T11:19:36.020 INFO:tasks.ceph.mgr.y.smithi084.stderr:2024-03-29T11:19:36.043+0000 7f45957a8040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-29T11:19:36.125 INFO:tasks.ceph.mgr.x.smithi179.stderr:2024-03-29T11:19:36.149+0000 7f8706237040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-29T11:19:36.177 INFO:tasks.ceph.mgr.x.smithi179.stderr:2024-03-29T11:19:36.200+0000 7f8706237040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-29T11:19:36.294 INFO:tasks.ceph.mgr.x.smithi179.stderr:2024-03-29T11:19:36.318+0000 7f8706237040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-29T11:19:36.295 INFO:tasks.ceph.mgr.y.smithi084.stderr:2024-03-29T11:19:36.319+0000 7f45957a8040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-29T11:19:36.352 INFO:tasks.ceph.mgr.y.smithi084.stderr:2024-03-29T11:19:36.375+0000 7f45957a8040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-29T11:19:36.470 INFO:tasks.ceph.mgr.y.smithi084.stderr:2024-03-29T11:19:36.493+0000 7f45957a8040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-29T11:19:37.308 INFO:tasks.ceph.mgr.x.smithi179.stderr:2024-03-29T11:19:37.331+0000 7f86a6f42640 -1 mgr.server handle_report got status from non-daemon mon.b 2024-03-29T11:19:37.308 INFO:tasks.ceph.mgr.x.smithi179.stderr:2024-03-29T11:19:37.331+0000 7f86a6f42640 -1 mgr.server handle_report got status from non-daemon mon.c 2024-03-29T11:19:38.256 INFO:tasks.ceph.osd.0.smithi084.stderr:2024-03-29T11:19:38.279+0000 7f36b7111640 -1 osd.0 0 log_to_monitors true 2024-03-29T11:19:38.279 INFO:tasks.ceph.osd.4.smithi179.stderr:2024-03-29T11:19:38.302+0000 7fc7d0d11640 -1 osd.4 0 log_to_monitors true 2024-03-29T11:19:38.309 INFO:tasks.ceph.osd.3.smithi084.stderr:2024-03-29T11:19:38.332+0000 7f56cfcf5640 -1 osd.3 0 log_to_monitors true 2024-03-29T11:19:38.331 INFO:tasks.ceph.osd.7.smithi179.stderr:2024-03-29T11:19:38.354+0000 7ffa32d21640 -1 osd.7 0 log_to_monitors true 2024-03-29T11:19:38.340 INFO:tasks.ceph.osd.5.smithi179.stderr:2024-03-29T11:19:38.364+0000 7fa54308b640 -1 osd.5 0 log_to_monitors true 2024-03-29T11:19:38.350 INFO:tasks.ceph.osd.1.smithi084.stderr:2024-03-29T11:19:38.374+0000 7fe314538640 -1 osd.1 0 log_to_monitors true 2024-03-29T11:19:38.351 INFO:tasks.ceph.osd.2.smithi084.stderr:2024-03-29T11:19:38.374+0000 7eff51a24640 -1 osd.2 0 log_to_monitors true 2024-03-29T11:19:38.389 INFO:tasks.ceph.osd.6.smithi179.stderr:2024-03-29T11:19:38.412+0000 7f5149822640 -1 osd.6 0 log_to_monitors true 2024-03-29T11:19:42.197 DEBUG:teuthology.orchestra.run.smithi084:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:19:42.350 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:42.373+0000 7f50e04f1640 10 monclient: get_monmap_and_config 2024-03-29T11:19:42.350 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:42.373+0000 7f50e04f1640 10 monclient: build_initial_monmap 2024-03-29T11:19:42.350 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:42.373+0000 7f50e04f1640 10 monclient: monmap: 2024-03-29T11:19:42.350 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-29T11:19:42.350 INFO:teuthology.misc.health.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:19:42.350 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T11:19:42.374555+0000 2024-03-29T11:19:42.350 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T11:19:42.374555+0000 2024-03-29T11:19:42.351 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-29T11:19:42.351 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:19:42.351 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-29T11:19:42.351 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.noname-c 2024-03-29T11:19:42.351 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-b 2024-03-29T11:19:42.351 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:19:42.352 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:42.374+0000 7f50e04f1640 1 Processor -- start 2024-03-29T11:19:42.352 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:42.374+0000 7f50e04f1640 1 -- start start 2024-03-29T11:19:42.352 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:42.374+0000 7f50e04f1640 10 monclient: init 2024-03-29T11:19:42.352 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:42.375+0000 7f50e04f1640 10 monclient: _reopen_session rank -1 2024-03-29T11:19:42.352 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:42.375+0000 7f50e04f1640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:19:42.352 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:42.375+0000 7f50e04f1640 10 monclient(hunting): picked mon.noname-a con 0x7f50d81422b0 addr v1:172.21.15.84:6789/0 2024-03-29T11:19:42.352 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:42.375+0000 7f50e04f1640 10 monclient(hunting): picked mon.noname-c con 0x7f50d8142690 addr v1:172.21.15.179:6789/0 2024-03-29T11:19:42.352 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:42.375+0000 7f50e04f1640 10 monclient(hunting): picked mon.noname-b con 0x7f50d80b17d0 addr v1:172.21.15.84:6790/0 2024-03-29T11:19:42.352 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:42.375+0000 7f50e04f1640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f50d8142c80 con 0x7f50d81422b0 2024-03-29T11:19:42.353 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:42.375+0000 7f50e04f1640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f50d80aae40 con 0x7f50d8142690 2024-03-29T11:19:42.353 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:42.375+0000 7f50e04f1640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f50d813fbd0 con 0x7f50d80b17d0 2024-03-29T11:19:42.353 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:42.375+0000 7f50e04f1640 10 monclient(hunting): _renew_subs 2024-03-29T11:19:42.353 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:42.375+0000 7f50e04f1640 10 monclient(hunting): authenticate will time out at 843.669833s 2024-03-29T11:19:42.353 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:42.376+0000 7f50dda8c640 1 --1- >> v1:172.21.15.179:6789/0 conn(0x7f50d8142690 0x7f50d80b2590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.179:6789/0 says I am v1:172.21.15.84:44214/0 (socket says 172.21.15.84:44214) 2024-03-29T11:19:42.353 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:42.376+0000 7f50dda8c640 1 -- 172.21.15.84:0/3024161538 learned_addr learned my addr 172.21.15.84:0/3024161538 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-29T11:19:42.355 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:42.376+0000 7f50dd28b640 1 -- 172.21.15.84:0/3024161538 <== mon.1 v1:172.21.15.179:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2327988761 0 0) 0x7f50d80aae40 con 0x7f50d8142690 2024-03-29T11:19:42.355 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:42.376+0000 7f50dd28b640 10 monclient(hunting): _negotiate 2024-03-29T11:19:42.355 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:42.376+0000 7f50dd28b640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:42.355 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:42.376+0000 7f50dd28b640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:42.355 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:42.376+0000 7f50dd28b640 10 monclient(hunting): my global_id is 4219 2024-03-29T11:19:42.355 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:42.376+0000 7f50dd28b640 1 -- 172.21.15.84:0/3024161538 --> v1:172.21.15.179:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f50c4003520 con 0x7f50d8142690 2024-03-29T11:19:42.355 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:42.376+0000 7f50dd28b640 1 -- 172.21.15.84:0/3024161538 <== mon.0 v1:172.21.15.84:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1237504807 0 0) 0x7f50d8142c80 con 0x7f50d81422b0 2024-03-29T11:19:42.355 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:42.376+0000 7f50dd28b640 10 monclient(hunting): _negotiate 2024-03-29T11:19:42.355 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:42.376+0000 7f50dd28b640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:42.356 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:42.376+0000 7f50dd28b640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:42.356 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:42.376+0000 7f50dd28b640 10 monclient(hunting): my global_id is 4242 2024-03-29T11:19:42.356 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:42.376+0000 7f50dd28b640 1 -- 172.21.15.84:0/3024161538 --> v1:172.21.15.84:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f50d80aae40 con 0x7f50d81422b0 2024-03-29T11:19:42.356 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:42.377+0000 7f50dd28b640 1 -- 172.21.15.84:0/3024161538 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 276086091 0 0) 0x7f50d813fbd0 con 0x7f50d80b17d0 2024-03-29T11:19:42.356 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:42.377+0000 7f50dd28b640 10 monclient(hunting): _negotiate 2024-03-29T11:19:42.356 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:42.377+0000 7f50dd28b640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:42.356 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:42.377+0000 7f50dd28b640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:42.356 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:42.377+0000 7f50dd28b640 10 monclient(hunting): my global_id is 4238 2024-03-29T11:19:42.356 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:42.377+0000 7f50dd28b640 1 -- 172.21.15.84:0/3024161538 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f50d8142c80 con 0x7f50d80b17d0 2024-03-29T11:19:42.356 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:42.377+0000 7f50dd28b640 1 -- 172.21.15.84:0/3024161538 <== mon.1 v1:172.21.15.179:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 470265594 0 0) 0x7f50c4003520 con 0x7f50d8142690 2024-03-29T11:19:42.356 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:42.377+0000 7f50dd28b640 1 -- 172.21.15.84:0/3024161538 >> v1:172.21.15.84:6790/0 conn(0x7f50d80b17d0 legacy=0x7f50d80b2ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:42.356 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:42.377+0000 7f50dd28b640 1 -- 172.21.15.84:0/3024161538 >> v1:172.21.15.84:6789/0 conn(0x7f50d81422b0 legacy=0x7f50d813f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:42.356 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:42.377+0000 7f50dd28b640 10 monclient: _finish_hunting 0 2024-03-29T11:19:42.356 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:42.377+0000 7f50dd28b640 1 monclient: found mon.noname-c 2024-03-29T11:19:42.357 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:42.377+0000 7f50dd28b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:19:42.357 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:42.377+0000 7f50dd28b640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.179:6789/0 2024-03-29T11:19:42.357 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:42.377+0000 7f50dd28b640 1 -- 172.21.15.84:0/3024161538 --> v1:172.21.15.179:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50d8151940 con 0x7f50d8142690 2024-03-29T11:19:42.357 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:42.377+0000 7f50dd28b640 10 monclient: _finish_auth 0 2024-03-29T11:19:42.357 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:42.377+0000 7f50dd28b640 10 monclient: _check_auth_tickets 2024-03-29T11:19:42.357 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:42.377+0000 7f50dd28b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:19:42.357 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:42.377+0000 7f50dd28b640 1 -- 172.21.15.84:0/3024161538 <== mon.1 v1:172.21.15.179:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4236868694 0 0) 0x7f50d40015c0 con 0x7f50d8142690 2024-03-29T11:19:42.357 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:42.377+0000 7f50e04f1640 5 monclient: authenticate success, global_id 4219 2024-03-29T11:19:42.357 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:42.377+0000 7f50e04f1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:19:42.357 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:42.377+0000 7f50dd28b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:42.357 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:42.377+0000 7f50dd28b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:19:42.357 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:42.377+0000 7f50dd28b640 10 monclient: dump: 2024-03-29T11:19:42.357 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T11:19:42.357 INFO:teuthology.misc.health.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:19:42.357 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T11:18:53.864025+0000 2024-03-29T11:19:42.358 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T11:18:53.864025+0000 2024-03-29T11:19:42.358 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:42.358 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:19:42.358 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T11:19:42.358 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T11:19:42.358 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T11:19:42.358 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:19:42.358 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:42.377+0000 7f50e04f1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:19:42.358 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:42.377+0000 7f50dd28b640 1 -- 172.21.15.84:0/3024161538 <== mon.1 v1:172.21.15.179:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f50d40017e0 con 0x7f50d8142690 2024-03-29T11:19:42.358 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:42.377+0000 7f50dd28b640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:19:42.358 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:42.377+0000 7f50e04f1640 10 monclient: get_monmap_and_config success 2024-03-29T11:19:42.358 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:42.377+0000 7f50e04f1640 4 set_mon_vals no callback set 2024-03-29T11:19:42.358 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:42.378+0000 7f50dd28b640 1 -- 172.21.15.84:0/3024161538 <== mon.1 v1:172.21.15.179:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4236868694 0 0) 0x7f50d4001af0 con 0x7f50d8142690 2024-03-29T11:19:42.358 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:42.378+0000 7f50dd28b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:42.358 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:42.378+0000 7f50dd28b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:19:42.359 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:42.378+0000 7f50dd28b640 10 monclient: dump: 2024-03-29T11:19:42.359 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T11:19:42.359 INFO:teuthology.misc.health.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:19:42.359 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T11:18:53.864025+0000 2024-03-29T11:19:42.359 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T11:18:53.864025+0000 2024-03-29T11:19:42.359 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:42.359 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:19:42.359 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T11:19:42.359 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T11:19:42.359 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T11:19:42.359 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:19:42.359 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:42.378+0000 7f50e04f1640 10 monclient: shutdown 2024-03-29T11:19:42.359 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:42.378+0000 7f50e04f1640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:19:42.359 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:42.378+0000 7f50e04f1640 1 -- 172.21.15.84:0/3024161538 >> v1:172.21.15.179:6789/0 conn(0x7f50d8142690 legacy=0x7f50d80b2590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:42.359 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:42.378+0000 7f50e04f1640 1 -- 172.21.15.84:0/3024161538 shutdown_connections 2024-03-29T11:19:42.359 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:42.378+0000 7f50e04f1640 1 -- 172.21.15.84:0/3024161538 >> 172.21.15.84:0/3024161538 conn(0x7f50d8066a00 msgr2=0x7f50d8066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:19:42.360 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:42.379+0000 7f50e04f1640 1 -- 172.21.15.84:0/3024161538 shutdown_connections 2024-03-29T11:19:42.366 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:42.379+0000 7f50e04f1640 1 -- 172.21.15.84:0/3024161538 wait complete. 2024-03-29T11:19:42.366 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:42.379+0000 7f50e04f1640 10 monclient: build_initial_monmap 2024-03-29T11:19:42.366 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:42.379+0000 7f50e04f1640 10 monclient: monmap: 2024-03-29T11:19:42.366 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-29T11:19:42.367 INFO:teuthology.misc.health.smithi084.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:19:42.367 INFO:teuthology.misc.health.smithi084.stderr:last_changed 0.000000 2024-03-29T11:19:42.367 INFO:teuthology.misc.health.smithi084.stderr:created 0.000000 2024-03-29T11:19:42.367 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-29T11:19:42.367 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:19:42.367 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-29T11:19:42.367 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.noname-b 2024-03-29T11:19:42.367 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-c 2024-03-29T11:19:42.367 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:19:42.367 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:42.380+0000 7f50e04f1640 1 librados: starting msgr at 2024-03-29T11:19:42.367 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:42.380+0000 7f50e04f1640 1 librados: starting objecter 2024-03-29T11:19:42.367 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:42.380+0000 7f50e04f1640 1 Processor -- start 2024-03-29T11:19:42.367 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:42.380+0000 7f50e04f1640 1 -- start start 2024-03-29T11:19:42.367 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:42.380+0000 7f50e04f1640 1 librados: setting wanted keys 2024-03-29T11:19:42.367 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:42.380+0000 7f50e04f1640 1 librados: calling monclient init 2024-03-29T11:19:42.368 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:42.380+0000 7f50e04f1640 10 monclient: init 2024-03-29T11:19:42.368 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:42.381+0000 7f50e04f1640 10 monclient: _reopen_session rank -1 2024-03-29T11:19:42.368 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:42.381+0000 7f50e04f1640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:19:42.368 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:42.381+0000 7f50e04f1640 10 monclient(hunting): picked mon.noname-c con 0x7f50d80b17d0 addr v1:172.21.15.84:6790/0 2024-03-29T11:19:42.368 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:42.381+0000 7f50e04f1640 10 monclient(hunting): picked mon.noname-a con 0x7f50d81422b0 addr v1:172.21.15.84:6789/0 2024-03-29T11:19:42.368 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:42.381+0000 7f50e04f1640 10 monclient(hunting): picked mon.noname-b con 0x7f50d8142690 addr v1:172.21.15.179:6789/0 2024-03-29T11:19:42.368 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:42.381+0000 7f50e04f1640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f50d80acd60 con 0x7f50d81422b0 2024-03-29T11:19:42.368 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:42.381+0000 7f50e04f1640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f50d80713f0 con 0x7f50d8142690 2024-03-29T11:19:42.368 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:42.381+0000 7f50e04f1640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f50d81e3f20 con 0x7f50d80b17d0 2024-03-29T11:19:42.368 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:42.381+0000 7f50e04f1640 10 monclient(hunting): _renew_subs 2024-03-29T11:19:42.368 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:42.381+0000 7f50e04f1640 10 monclient(hunting): authenticate will time out at 843.675399s 2024-03-29T11:19:42.368 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:42.381+0000 7f50dea8e640 1 --1- >> v1:172.21.15.179:6789/0 conn(0x7f50d8142690 0x7f50d813bd10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.179:6789/0 says I am v1:172.21.15.84:44358/0 (socket says 172.21.15.84:44358) 2024-03-29T11:19:42.368 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:42.381+0000 7f50dea8e640 1 -- 172.21.15.84:0/2674636442 learned_addr learned my addr 172.21.15.84:0/2674636442 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-29T11:19:42.368 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:42.382+0000 7f50caffd640 10 client.?.objecter ms_handle_connect 0x7f50d8142690 2024-03-29T11:19:42.368 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:42.382+0000 7f50caffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:19:42.369 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:42.382+0000 7f50caffd640 10 client.?.objecter ms_handle_connect 0x7f50d80b17d0 2024-03-29T11:19:42.369 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:42.382+0000 7f50caffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:19:42.369 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:42.382+0000 7f50caffd640 10 client.?.objecter ms_handle_connect 0x7f50d81422b0 2024-03-29T11:19:42.369 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:42.382+0000 7f50caffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:19:42.369 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:42.382+0000 7f50caffd640 1 -- 172.21.15.84:0/2674636442 <== mon.1 v1:172.21.15.179:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 495565557 0 0) 0x7f50d80713f0 con 0x7f50d8142690 2024-03-29T11:19:42.369 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:42.382+0000 7f50caffd640 10 monclient(hunting): _negotiate 2024-03-29T11:19:42.369 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:42.382+0000 7f50caffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:42.369 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:42.382+0000 7f50caffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:42.369 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:42.382+0000 7f50caffd640 10 monclient(hunting): my global_id is 4222 2024-03-29T11:19:42.369 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:42.382+0000 7f50caffd640 1 -- 172.21.15.84:0/2674636442 --> v1:172.21.15.179:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f50bc004eb0 con 0x7f50d8142690 2024-03-29T11:19:42.369 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:42.382+0000 7f50caffd640 1 -- 172.21.15.84:0/2674636442 <== mon.0 v1:172.21.15.84:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3114156321 0 0) 0x7f50d80acd60 con 0x7f50d81422b0 2024-03-29T11:19:42.369 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:42.382+0000 7f50caffd640 10 monclient(hunting): _negotiate 2024-03-29T11:19:42.369 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:42.382+0000 7f50caffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:42.369 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:42.382+0000 7f50caffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:42.369 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:42.382+0000 7f50caffd640 10 monclient(hunting): my global_id is 4245 2024-03-29T11:19:42.370 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:42.382+0000 7f50caffd640 1 -- 172.21.15.84:0/2674636442 --> v1:172.21.15.84:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f50d80713f0 con 0x7f50d81422b0 2024-03-29T11:19:42.370 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:42.383+0000 7f50caffd640 1 -- 172.21.15.84:0/2674636442 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3572201855 0 0) 0x7f50d81e3f20 con 0x7f50d80b17d0 2024-03-29T11:19:42.370 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:42.383+0000 7f50caffd640 10 monclient(hunting): _negotiate 2024-03-29T11:19:42.370 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:42.383+0000 7f50caffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:42.370 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:42.383+0000 7f50caffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:42.370 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:42.383+0000 7f50caffd640 10 monclient(hunting): my global_id is 4241 2024-03-29T11:19:42.370 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:42.383+0000 7f50caffd640 1 -- 172.21.15.84:0/2674636442 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f50d80acd60 con 0x7f50d80b17d0 2024-03-29T11:19:42.370 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:42.383+0000 7f50caffd640 1 -- 172.21.15.84:0/2674636442 <== mon.1 v1:172.21.15.179:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3314143560 0 0) 0x7f50bc004eb0 con 0x7f50d8142690 2024-03-29T11:19:42.370 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:42.383+0000 7f50caffd640 1 -- 172.21.15.84:0/2674636442 --> v1:172.21.15.179:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f50bc005670 con 0x7f50d8142690 2024-03-29T11:19:42.370 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:42.383+0000 7f50caffd640 1 -- 172.21.15.84:0/2674636442 <== mon.1 v1:172.21.15.179:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4236868694 0 0) 0x7f50d0001d10 con 0x7f50d8142690 2024-03-29T11:19:42.370 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:42.383+0000 7f50caffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:42.370 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:42.383+0000 7f50caffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:19:42.370 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:42.383+0000 7f50caffd640 10 monclient(hunting): dump: 2024-03-29T11:19:42.370 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T11:19:42.370 INFO:teuthology.misc.health.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:19:42.371 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T11:18:53.864025+0000 2024-03-29T11:19:42.371 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T11:18:53.864025+0000 2024-03-29T11:19:42.371 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:42.371 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:19:42.371 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T11:19:42.371 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T11:19:42.371 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T11:19:42.371 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:19:42.371 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:42.383+0000 7f50caffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:19:42.371 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:42.383+0000 7f50caffd640 1 -- 172.21.15.84:0/2674636442 <== mon.2 v1:172.21.15.84:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1815582174 0 0) 0x7f50d80acd60 con 0x7f50d80b17d0 2024-03-29T11:19:42.371 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:42.383+0000 7f50caffd640 1 -- 172.21.15.84:0/2674636442 --> v1:172.21.15.84:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f50bc004eb0 con 0x7f50d80b17d0 2024-03-29T11:19:42.371 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:42.383+0000 7f50caffd640 1 -- 172.21.15.84:0/2674636442 <== mon.1 v1:172.21.15.179:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2560042044 0 0) 0x7f50bc005670 con 0x7f50d8142690 2024-03-29T11:19:42.371 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:42.383+0000 7f50caffd640 1 -- 172.21.15.84:0/2674636442 >> v1:172.21.15.84:6790/0 conn(0x7f50d80b17d0 legacy=0x7f50d8071b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:42.371 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:42.383+0000 7f50caffd640 1 -- 172.21.15.84:0/2674636442 >> v1:172.21.15.84:6789/0 conn(0x7f50d81422b0 legacy=0x7f50d81e4440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:42.371 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:42.384+0000 7f50caffd640 10 monclient: _finish_hunting 0 2024-03-29T11:19:42.371 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:42.384+0000 7f50caffd640 1 monclient: found mon.b 2024-03-29T11:19:42.371 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:42.384+0000 7f50caffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:19:42.372 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:42.384+0000 7f50caffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-29T11:19:42.372 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:42.384+0000 7f50caffd640 1 -- 172.21.15.84:0/2674636442 --> v1:172.21.15.179:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50d81e4b50 con 0x7f50d8142690 2024-03-29T11:19:42.372 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:42.384+0000 7f50caffd640 10 monclient: _finish_auth 0 2024-03-29T11:19:42.372 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:42.384+0000 7f50caffd640 10 monclient: _check_auth_tickets 2024-03-29T11:19:42.372 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:42.384+0000 7f50caffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:19:42.372 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:42.384+0000 7f50e04f1640 5 monclient: authenticate success, global_id 4222 2024-03-29T11:19:42.372 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:42.384+0000 7f50e04f1640 10 monclient: _renew_subs 2024-03-29T11:19:42.372 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:42.384+0000 7f50e04f1640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-29T11:19:42.372 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:42.384+0000 7f50e04f1640 1 -- 172.21.15.84:0/2674636442 --> v1:172.21.15.179:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f50d806eee0 con 0x7f50d8142690 2024-03-29T11:19:42.372 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:42.384+0000 7f50e04f1640 10 client.4222.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:19:42.372 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:42.384+0000 7f50e04f1640 10 monclient: _renew_subs 2024-03-29T11:19:42.372 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:42.384+0000 7f50e04f1640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-29T11:19:42.372 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:42.384+0000 7f50e04f1640 1 -- 172.21.15.84:0/2674636442 --> v1:172.21.15.179:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f50d813c600 con 0x7f50d8142690 2024-03-29T11:19:42.372 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:42.384+0000 7f50e04f1640 1 librados: init done 2024-03-29T11:19:42.373 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:42.384+0000 7f50caffd640 1 -- 172.21.15.84:0/2674636442 <== mon.1 v1:172.21.15.179:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f50d0003990 con 0x7f50d8142690 2024-03-29T11:19:42.373 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:42.384+0000 7f50caffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:19:42.373 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:42.384+0000 7f50caffd640 1 -- 172.21.15.84:0/2674636442 <== mon.1 v1:172.21.15.179:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4236868694 0 0) 0x7f50d0003d20 con 0x7f50d8142690 2024-03-29T11:19:42.373 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:42.384+0000 7f50caffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:42.373 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:42.384+0000 7f50caffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:19:42.373 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:42.384+0000 7f50caffd640 10 monclient: dump: 2024-03-29T11:19:42.373 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T11:19:42.373 INFO:teuthology.misc.health.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:19:42.373 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T11:18:53.864025+0000 2024-03-29T11:19:42.373 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T11:18:53.864025+0000 2024-03-29T11:19:42.373 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:42.373 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:19:42.373 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T11:19:42.373 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T11:19:42.373 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T11:19:42.373 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:19:42.373 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:42.385+0000 7f50dca8a640 4 set_mon_vals no callback set 2024-03-29T11:19:42.374 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:42.385+0000 7f50e04f1640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:42.374 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:42.386+0000 7f50e04f1640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:42.374 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:42.386+0000 7f50e04f1640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-29T11:19:42.374 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:42.386+0000 7f50e04f1640 1 -- 172.21.15.84:0/2674636442 --> v1:172.21.15.179:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f50c0005080 con 0x7f50d8142690 2024-03-29T11:19:42.374 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:42.387+0000 7f50caffd640 1 -- 172.21.15.84:0/2674636442 <== mon.1 v1:172.21.15.179:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4033942909 0 0) 0x7f50d0003e80 con 0x7f50d8142690 2024-03-29T11:19:42.374 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:42.388+0000 7f50caffd640 1 -- 172.21.15.84:0/2674636442 <== mon.1 v1:172.21.15.179:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2808139587 0 0) 0x7f50d0084d70 con 0x7f50d8142690 2024-03-29T11:19:42.374 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:42.388+0000 7f50caffd640 10 client.4222.objecter ms_dispatch 0x7f50d8000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:19:42.374 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:42.388+0000 7f50caffd640 3 client.4222.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:19:42.374 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:42.388+0000 7f50caffd640 3 client.4222.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:19:42.374 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:42.388+0000 7f50caffd640 20 client.4222.objecter dump_active .. 0 homeless 2024-03-29T11:19:42.374 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:42.388+0000 7f50de28d640 10 monclient: get_auth_request con 0x7f50bc06e070 auth_method 0 2024-03-29T11:19:42.374 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:42.389+0000 7f50caffd640 10 client.4222.objecter ms_handle_connect 0x7f50bc06e070 2024-03-29T11:19:42.375 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:42.394+0000 7f50caffd640 1 -- 172.21.15.84:0/2674636442 <== mon.1 v1:172.21.15.179:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f50d0050650 con 0x7f50d8142690 2024-03-29T11:19:42.375 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:42.394+0000 7f50caffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:42.375 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:42.394+0000 7f50caffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:19:42.743 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:42.766+0000 7f50e04f1640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:19:42.743 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:42.766+0000 7f50e04f1640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:19:42.743 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:42.766+0000 7f50e04f1640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-29T11:19:42.743 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:42.766+0000 7f50e04f1640 1 -- 172.21.15.84:0/2674636442 --> v1:172.21.15.179:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f50c0005330 con 0x7f50d8142690 2024-03-29T11:19:42.743 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:42.767+0000 7f50caffd640 1 -- 172.21.15.84:0/2674636442 <== mon.1 v1:172.21.15.179:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4209477549) 0x7f50d0052580 con 0x7f50d8142690 2024-03-29T11:19:42.744 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:42.767+0000 7f50caffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:19:42.744 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:42.767+0000 7f50caffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:19:42.744 INFO:teuthology.misc.health.smithi084.stdout: 2024-03-29T11:19:42.744 INFO:teuthology.misc.health.smithi084.stdout:{"epoch":10,"fsid":"2c3e002f-9eb0-4860-853b-4ef1ef562213","created":"2024-03-29T11:19:32.029169+0000","modified":"2024-03-29T11:19:34.200299+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:34.200299+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"5c39e276-2034-4716-bbc0-57a634eefa92","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"cd580e99-109f-4bf1-88c0-a5987bc8b9ed","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"11e60a5a-051d-453f-a190-66387cb35592","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"2c329fbe-bbc4-4030-bf87-d711cad04679","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f4213d4a-bd36-4433-91ae-d7e2b02481d3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"f0cf44ea-6055-48af-805b-aed858102ff9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"d895eaa0-cc0b-4ee4-9b5a-2dfcd89a60b7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"9bc5a7bc-8f52-4db9-96d6-de5e68d52eaa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:19:42.747 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:42.771+0000 7f50e04f1640 10 librados: watch_flush enter 2024-03-29T11:19:42.747 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:42.771+0000 7f50e04f1640 10 librados: watch_flush exit 2024-03-29T11:19:42.748 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:42.771+0000 7f50e04f1640 20 client.4222.objecter shutdown clearing up homeless session... 2024-03-29T11:19:42.748 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:42.771+0000 7f50e04f1640 10 client.4222.objecter successfully canceled tick 2024-03-29T11:19:42.748 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:42.771+0000 7f50e04f1640 1 -- 172.21.15.84:0/2674636442 >> v1:172.21.15.179:6816/2981134980 conn(0x7f50bc06e070 legacy=0x7f50bc070530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:42.748 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:42.771+0000 7f50e04f1640 10 monclient: shutdown 2024-03-29T11:19:42.748 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:42.771+0000 7f50e04f1640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:19:42.748 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:42.771+0000 7f50e04f1640 1 -- 172.21.15.84:0/2674636442 >> v1:172.21.15.179:6789/0 conn(0x7f50d8142690 legacy=0x7f50d813bd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:42.748 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:42.771+0000 7f50e04f1640 1 -- 172.21.15.84:0/2674636442 shutdown_connections 2024-03-29T11:19:42.748 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:42.771+0000 7f50e04f1640 1 -- 172.21.15.84:0/2674636442 >> 172.21.15.84:0/2674636442 conn(0x7f50d8066a00 msgr2=0x7f50d81416b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:19:42.749 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:42.771+0000 7f50e04f1640 1 -- 172.21.15.84:0/2674636442 shutdown_connections 2024-03-29T11:19:42.749 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:42.772+0000 7f50e04f1640 1 -- 172.21.15.84:0/2674636442 wait complete. 2024-03-29T11:19:42.749 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:42.772+0000 7f50e04f1640 1 librados: shutdown 2024-03-29T11:19:42.762 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:19:49.569 DEBUG:teuthology.orchestra.run.smithi084:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:19:49.712 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:49.735+0000 7f6b9bda5640 10 monclient: get_monmap_and_config 2024-03-29T11:19:49.712 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:49.735+0000 7f6b9bda5640 10 monclient: build_initial_monmap 2024-03-29T11:19:49.712 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:49.735+0000 7f6b9bda5640 10 monclient: monmap: 2024-03-29T11:19:49.712 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-29T11:19:49.712 INFO:teuthology.misc.health.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:19:49.712 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T11:19:49.736280+0000 2024-03-29T11:19:49.712 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T11:19:49.736280+0000 2024-03-29T11:19:49.712 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-29T11:19:49.712 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:19:49.712 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-29T11:19:49.713 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.noname-c 2024-03-29T11:19:49.713 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-b 2024-03-29T11:19:49.713 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:19:49.725 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:49.736+0000 7f6b9bda5640 1 Processor -- start 2024-03-29T11:19:49.725 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:49.736+0000 7f6b9bda5640 1 -- start start 2024-03-29T11:19:49.726 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:49.736+0000 7f6b9bda5640 10 monclient: init 2024-03-29T11:19:49.726 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:49.737+0000 7f6b9bda5640 10 monclient: _reopen_session rank -1 2024-03-29T11:19:49.726 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:49.737+0000 7f6b9bda5640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:19:49.726 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:49.737+0000 7f6b9bda5640 10 monclient(hunting): picked mon.noname-c con 0x7f6b941457f0 addr v1:172.21.15.179:6789/0 2024-03-29T11:19:49.726 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:49.737+0000 7f6b9bda5640 10 monclient(hunting): picked mon.noname-a con 0x7f6b9413f650 addr v1:172.21.15.84:6789/0 2024-03-29T11:19:49.726 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:49.737+0000 7f6b9bda5640 10 monclient(hunting): picked mon.noname-b con 0x7f6b940a9d70 addr v1:172.21.15.84:6790/0 2024-03-29T11:19:49.726 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:49.737+0000 7f6b9bda5640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6b940aaef0 con 0x7f6b9413f650 2024-03-29T11:19:49.726 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:49.737+0000 7f6b9bda5640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6b94145bd0 con 0x7f6b941457f0 2024-03-29T11:19:49.726 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:49.737+0000 7f6b9bda5640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6b94145eb0 con 0x7f6b940a9d70 2024-03-29T11:19:49.726 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:49.737+0000 7f6b9bda5640 10 monclient(hunting): _renew_subs 2024-03-29T11:19:49.726 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:49.737+0000 7f6b9bda5640 10 monclient(hunting): authenticate will time out at 851.031523s 2024-03-29T11:19:49.726 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:49.737+0000 7f6b99b41640 1 --1- >> v1:172.21.15.179:6789/0 conn(0x7f6b941457f0 0x7f6b9413ef40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.179:6789/0 says I am v1:172.21.15.84:36024/0 (socket says 172.21.15.84:36024) 2024-03-29T11:19:49.726 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:49.737+0000 7f6b99b41640 1 -- 172.21.15.84:0/3747730656 learned_addr learned my addr 172.21.15.84:0/3747730656 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-29T11:19:49.727 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:49.737+0000 7f6b98b3f640 1 -- 172.21.15.84:0/3747730656 <== mon.0 v1:172.21.15.84:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2275387226 0 0) 0x7f6b940aaef0 con 0x7f6b9413f650 2024-03-29T11:19:49.727 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:49.737+0000 7f6b98b3f640 10 monclient(hunting): _negotiate 2024-03-29T11:19:49.727 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:49.737+0000 7f6b98b3f640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:49.727 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:49.737+0000 7f6b98b3f640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:49.727 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:49.737+0000 7f6b98b3f640 10 monclient(hunting): my global_id is 4248 2024-03-29T11:19:49.727 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:49.737+0000 7f6b98b3f640 1 -- 172.21.15.84:0/3747730656 --> v1:172.21.15.84:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6b80003520 con 0x7f6b9413f650 2024-03-29T11:19:49.727 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:49.738+0000 7f6b98b3f640 1 -- 172.21.15.84:0/3747730656 <== mon.1 v1:172.21.15.179:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1756756828 0 0) 0x7f6b94145bd0 con 0x7f6b941457f0 2024-03-29T11:19:49.727 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:49.738+0000 7f6b98b3f640 10 monclient(hunting): _negotiate 2024-03-29T11:19:49.727 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:49.738+0000 7f6b98b3f640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:49.727 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:49.738+0000 7f6b98b3f640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:49.727 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:49.738+0000 7f6b98b3f640 10 monclient(hunting): my global_id is 4225 2024-03-29T11:19:49.727 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:49.738+0000 7f6b98b3f640 1 -- 172.21.15.84:0/3747730656 --> v1:172.21.15.179:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6b940aaef0 con 0x7f6b941457f0 2024-03-29T11:19:49.727 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:49.738+0000 7f6b98b3f640 1 -- 172.21.15.84:0/3747730656 <== mon.0 v1:172.21.15.84:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 865543692 0 0) 0x7f6b80003520 con 0x7f6b9413f650 2024-03-29T11:19:49.727 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:49.738+0000 7f6b98b3f640 1 -- 172.21.15.84:0/3747730656 >> v1:172.21.15.84:6790/0 conn(0x7f6b940a9d70 legacy=0x7f6b9414d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:49.727 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:49.738+0000 7f6b98b3f640 1 -- 172.21.15.84:0/3747730656 >> v1:172.21.15.179:6789/0 conn(0x7f6b941457f0 legacy=0x7f6b9413ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:49.728 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:49.738+0000 7f6b98b3f640 10 monclient: _finish_hunting 0 2024-03-29T11:19:49.728 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:49.738+0000 7f6b98b3f640 1 monclient: found mon.noname-a 2024-03-29T11:19:49.728 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:49.738+0000 7f6b98b3f640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:19:49.728 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:49.738+0000 7f6b98b3f640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.84:6789/0 2024-03-29T11:19:49.728 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:49.738+0000 7f6b98b3f640 1 -- 172.21.15.84:0/3747730656 --> v1:172.21.15.84:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b94146340 con 0x7f6b9413f650 2024-03-29T11:19:49.728 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:49.738+0000 7f6b98b3f640 10 monclient: _finish_auth 0 2024-03-29T11:19:49.728 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:49.738+0000 7f6b98b3f640 10 monclient: _check_auth_tickets 2024-03-29T11:19:49.728 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:49.738+0000 7f6b98b3f640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:19:49.728 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:49.738+0000 7f6b98b3f640 1 -- 172.21.15.84:0/3747730656 <== mon.0 v1:172.21.15.84:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4236868694 0 0) 0x7f6b90001690 con 0x7f6b9413f650 2024-03-29T11:19:49.728 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:49.738+0000 7f6b98b3f640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:49.728 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:49.738+0000 7f6b98b3f640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:19:49.728 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:49.738+0000 7f6b98b3f640 10 monclient: dump: 2024-03-29T11:19:49.728 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T11:19:49.728 INFO:teuthology.misc.health.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:19:49.728 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T11:18:53.864025+0000 2024-03-29T11:19:49.729 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T11:18:53.864025+0000 2024-03-29T11:19:49.729 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:49.729 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:19:49.729 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T11:19:49.729 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T11:19:49.729 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T11:19:49.729 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:19:49.729 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:49.738+0000 7f6b9bda5640 5 monclient: authenticate success, global_id 4248 2024-03-29T11:19:49.729 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:49.738+0000 7f6b9bda5640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:19:49.729 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:49.739+0000 7f6b98b3f640 1 -- 172.21.15.84:0/3747730656 <== mon.0 v1:172.21.15.84:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6b900017f0 con 0x7f6b9413f650 2024-03-29T11:19:49.729 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:49.739+0000 7f6b98b3f640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:19:49.729 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:49.739+0000 7f6b98b3f640 1 -- 172.21.15.84:0/3747730656 <== mon.0 v1:172.21.15.84:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4236868694 0 0) 0x7f6b90001c00 con 0x7f6b9413f650 2024-03-29T11:19:49.729 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:49.739+0000 7f6b9bda5640 10 monclient: get_monmap_and_config success 2024-03-29T11:19:49.729 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:49.739+0000 7f6b98b3f640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:49.729 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:49.739+0000 7f6b98b3f640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:19:49.730 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:49.739+0000 7f6b98b3f640 10 monclient: dump: 2024-03-29T11:19:49.730 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T11:19:49.730 INFO:teuthology.misc.health.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:19:49.730 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T11:18:53.864025+0000 2024-03-29T11:19:49.730 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T11:18:53.864025+0000 2024-03-29T11:19:49.730 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:49.730 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:19:49.730 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T11:19:49.730 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T11:19:49.730 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T11:19:49.730 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:19:49.730 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:49.739+0000 7f6b9bda5640 4 set_mon_vals no callback set 2024-03-29T11:19:49.730 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:49.739+0000 7f6b9bda5640 10 monclient: shutdown 2024-03-29T11:19:49.730 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:49.739+0000 7f6b9bda5640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:19:49.731 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:49.739+0000 7f6b9bda5640 1 -- 172.21.15.84:0/3747730656 >> v1:172.21.15.84:6789/0 conn(0x7f6b9413f650 legacy=0x7f6b940b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:49.731 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:49.739+0000 7f6b9bda5640 1 -- 172.21.15.84:0/3747730656 shutdown_connections 2024-03-29T11:19:49.731 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:49.739+0000 7f6b9bda5640 1 -- 172.21.15.84:0/3747730656 >> 172.21.15.84:0/3747730656 conn(0x7f6b94066a00 msgr2=0x7f6b94066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:19:49.731 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:49.740+0000 7f6b9bda5640 1 -- 172.21.15.84:0/3747730656 shutdown_connections 2024-03-29T11:19:49.731 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:49.740+0000 7f6b9bda5640 1 -- 172.21.15.84:0/3747730656 wait complete. 2024-03-29T11:19:49.731 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:49.740+0000 7f6b9bda5640 10 monclient: build_initial_monmap 2024-03-29T11:19:49.731 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:49.740+0000 7f6b9bda5640 10 monclient: monmap: 2024-03-29T11:19:49.731 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-29T11:19:49.731 INFO:teuthology.misc.health.smithi084.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:19:49.731 INFO:teuthology.misc.health.smithi084.stderr:last_changed 0.000000 2024-03-29T11:19:49.731 INFO:teuthology.misc.health.smithi084.stderr:created 0.000000 2024-03-29T11:19:49.731 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-29T11:19:49.731 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:19:49.731 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-29T11:19:49.731 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.noname-b 2024-03-29T11:19:49.732 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-c 2024-03-29T11:19:49.732 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:19:49.732 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:49.741+0000 7f6b9bda5640 1 librados: starting msgr at 2024-03-29T11:19:49.732 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:49.741+0000 7f6b9bda5640 1 librados: starting objecter 2024-03-29T11:19:49.732 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:49.741+0000 7f6b9bda5640 1 Processor -- start 2024-03-29T11:19:49.732 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:49.741+0000 7f6b9bda5640 1 -- start start 2024-03-29T11:19:49.732 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:49.741+0000 7f6b9bda5640 1 librados: setting wanted keys 2024-03-29T11:19:49.732 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:49.741+0000 7f6b9bda5640 1 librados: calling monclient init 2024-03-29T11:19:49.732 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:49.741+0000 7f6b9bda5640 10 monclient: init 2024-03-29T11:19:49.732 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:49.742+0000 7f6b9bda5640 10 monclient: _reopen_session rank -1 2024-03-29T11:19:49.732 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:49.742+0000 7f6b9bda5640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:19:49.732 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:49.742+0000 7f6b9bda5640 10 monclient(hunting): picked mon.noname-b con 0x7f6b940a9d70 addr v1:172.21.15.179:6789/0 2024-03-29T11:19:49.732 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:49.742+0000 7f6b9bda5640 10 monclient(hunting): picked mon.noname-c con 0x7f6b9413f650 addr v1:172.21.15.84:6790/0 2024-03-29T11:19:49.733 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:49.742+0000 7f6b9bda5640 10 monclient(hunting): picked mon.noname-a con 0x7f6b941457f0 addr v1:172.21.15.84:6789/0 2024-03-29T11:19:49.733 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:49.742+0000 7f6b9bda5640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6b9413bda0 con 0x7f6b941457f0 2024-03-29T11:19:49.733 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:49.742+0000 7f6b9bda5640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6b941cbbb0 con 0x7f6b940a9d70 2024-03-29T11:19:49.733 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:49.742+0000 7f6b9bda5640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6b941cefc0 con 0x7f6b9413f650 2024-03-29T11:19:49.733 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:49.742+0000 7f6b9bda5640 10 monclient(hunting): _renew_subs 2024-03-29T11:19:49.733 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:49.742+0000 7f6b9bda5640 10 monclient(hunting): authenticate will time out at 851.036542s 2024-03-29T11:19:49.733 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:49.742+0000 7f6b99340640 1 --1- >> v1:172.21.15.84:6790/0 conn(0x7f6b9413f650 0x7f6b941d7e00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.84:6790/0 says I am v1:172.21.15.84:51516/0 (socket says 172.21.15.84:51516) 2024-03-29T11:19:49.733 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:49.742+0000 7f6b99340640 1 -- 172.21.15.84:0/3076583940 learned_addr learned my addr 172.21.15.84:0/3076583940 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-29T11:19:49.733 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:49.742+0000 7f6b867fc640 10 client.?.objecter ms_handle_connect 0x7f6b940a9d70 2024-03-29T11:19:49.733 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:49.742+0000 7f6b867fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:19:49.733 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:49.743+0000 7f6b867fc640 10 client.?.objecter ms_handle_connect 0x7f6b9413f650 2024-03-29T11:19:49.733 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:49.743+0000 7f6b867fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:19:49.733 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:49.743+0000 7f6b867fc640 10 client.?.objecter ms_handle_connect 0x7f6b941457f0 2024-03-29T11:19:49.733 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:49.743+0000 7f6b867fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:19:49.734 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:49.743+0000 7f6b867fc640 1 -- 172.21.15.84:0/3076583940 <== mon.1 v1:172.21.15.179:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 492115930 0 0) 0x7f6b941cbbb0 con 0x7f6b940a9d70 2024-03-29T11:19:49.734 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:49.743+0000 7f6b867fc640 10 monclient(hunting): _negotiate 2024-03-29T11:19:49.734 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:49.743+0000 7f6b867fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:49.734 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:49.743+0000 7f6b867fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:49.734 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:49.743+0000 7f6b867fc640 10 monclient(hunting): my global_id is 4228 2024-03-29T11:19:49.734 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:49.743+0000 7f6b867fc640 1 -- 172.21.15.84:0/3076583940 --> v1:172.21.15.179:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6b78004eb0 con 0x7f6b940a9d70 2024-03-29T11:19:49.734 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:49.743+0000 7f6b867fc640 1 -- 172.21.15.84:0/3076583940 <== mon.0 v1:172.21.15.84:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 702873696 0 0) 0x7f6b9413bda0 con 0x7f6b941457f0 2024-03-29T11:19:49.734 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:49.743+0000 7f6b867fc640 10 monclient(hunting): _negotiate 2024-03-29T11:19:49.734 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:49.743+0000 7f6b867fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:49.734 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:49.743+0000 7f6b867fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:49.734 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:49.743+0000 7f6b867fc640 10 monclient(hunting): my global_id is 4251 2024-03-29T11:19:49.734 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:49.743+0000 7f6b867fc640 1 -- 172.21.15.84:0/3076583940 --> v1:172.21.15.84:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6b941cbbb0 con 0x7f6b941457f0 2024-03-29T11:19:49.734 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:49.743+0000 7f6b867fc640 1 -- 172.21.15.84:0/3076583940 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3752577133 0 0) 0x7f6b941cefc0 con 0x7f6b9413f650 2024-03-29T11:19:49.734 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:49.743+0000 7f6b867fc640 10 monclient(hunting): _negotiate 2024-03-29T11:19:49.734 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:49.743+0000 7f6b867fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:49.735 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:49.743+0000 7f6b867fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:49.735 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:49.743+0000 7f6b867fc640 10 monclient(hunting): my global_id is 4247 2024-03-29T11:19:49.735 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:49.743+0000 7f6b867fc640 1 -- 172.21.15.84:0/3076583940 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6b9413bda0 con 0x7f6b9413f650 2024-03-29T11:19:49.735 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:49.744+0000 7f6b867fc640 1 -- 172.21.15.84:0/3076583940 <== mon.1 v1:172.21.15.179:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2387034153 0 0) 0x7f6b78004eb0 con 0x7f6b940a9d70 2024-03-29T11:19:49.735 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:49.744+0000 7f6b867fc640 1 -- 172.21.15.84:0/3076583940 --> v1:172.21.15.179:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6b941cefc0 con 0x7f6b940a9d70 2024-03-29T11:19:49.735 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:49.744+0000 7f6b867fc640 1 -- 172.21.15.84:0/3076583940 <== mon.1 v1:172.21.15.179:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4236868694 0 0) 0x7f6b880018b0 con 0x7f6b940a9d70 2024-03-29T11:19:49.735 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:49.744+0000 7f6b867fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:49.735 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:49.744+0000 7f6b867fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:19:49.735 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:49.744+0000 7f6b867fc640 10 monclient(hunting): dump: 2024-03-29T11:19:49.735 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T11:19:49.735 INFO:teuthology.misc.health.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:19:49.735 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T11:18:53.864025+0000 2024-03-29T11:19:49.735 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T11:18:53.864025+0000 2024-03-29T11:19:49.735 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:49.735 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:19:49.736 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T11:19:49.736 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T11:19:49.736 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T11:19:49.736 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:19:49.736 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:49.744+0000 7f6b867fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:19:49.736 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:49.744+0000 7f6b867fc640 1 -- 172.21.15.84:0/3076583940 <== mon.2 v1:172.21.15.84:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3304433733 0 0) 0x7f6b9413bda0 con 0x7f6b9413f650 2024-03-29T11:19:49.736 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:49.744+0000 7f6b867fc640 1 -- 172.21.15.84:0/3076583940 --> v1:172.21.15.84:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6b78005b80 con 0x7f6b9413f650 2024-03-29T11:19:49.736 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:49.744+0000 7f6b867fc640 1 -- 172.21.15.84:0/3076583940 <== mon.2 v1:172.21.15.84:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4236868694 0 0) 0x7f6b90001b00 con 0x7f6b9413f650 2024-03-29T11:19:49.736 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:49.744+0000 7f6b867fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:49.736 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:49.744+0000 7f6b867fc640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:19:49.736 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:49.744+0000 7f6b867fc640 10 monclient(hunting): dump: 2024-03-29T11:19:49.736 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T11:19:49.736 INFO:teuthology.misc.health.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:19:49.736 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T11:18:53.864025+0000 2024-03-29T11:19:49.736 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T11:18:53.864025+0000 2024-03-29T11:19:49.737 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:49.737 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:19:49.737 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T11:19:49.737 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T11:19:49.737 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T11:19:49.737 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:19:49.737 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:49.744+0000 7f6b867fc640 1 -- 172.21.15.84:0/3076583940 <== mon.0 v1:172.21.15.84:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3124858557 0 0) 0x7f6b941cbbb0 con 0x7f6b941457f0 2024-03-29T11:19:49.737 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:49.744+0000 7f6b867fc640 1 -- 172.21.15.84:0/3076583940 --> v1:172.21.15.84:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6b9413bda0 con 0x7f6b941457f0 2024-03-29T11:19:49.737 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:49.744+0000 7f6b867fc640 1 -- 172.21.15.84:0/3076583940 <== mon.0 v1:172.21.15.84:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4236868694 0 0) 0x7f6b8c001b30 con 0x7f6b941457f0 2024-03-29T11:19:49.737 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:49.744+0000 7f6b867fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:49.737 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:49.744+0000 7f6b867fc640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:19:49.737 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:49.744+0000 7f6b867fc640 10 monclient(hunting): dump: 2024-03-29T11:19:49.737 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T11:19:49.737 INFO:teuthology.misc.health.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:19:49.738 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T11:18:53.864025+0000 2024-03-29T11:19:49.738 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T11:18:53.864025+0000 2024-03-29T11:19:49.738 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:49.738 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:19:49.738 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T11:19:49.738 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T11:19:49.738 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T11:19:49.738 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:19:49.738 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:49.744+0000 7f6b867fc640 1 -- 172.21.15.84:0/3076583940 <== mon.2 v1:172.21.15.84:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2445866414 0 0) 0x7f6b78005b80 con 0x7f6b9413f650 2024-03-29T11:19:49.738 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:49.744+0000 7f6b867fc640 1 -- 172.21.15.84:0/3076583940 >> v1:172.21.15.179:6789/0 conn(0x7f6b940a9d70 legacy=0x7f6b941d76f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:49.738 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:49.744+0000 7f6b867fc640 1 -- 172.21.15.84:0/3076583940 >> v1:172.21.15.84:6789/0 conn(0x7f6b941457f0 legacy=0x7f6b941e2130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:49.738 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:49.744+0000 7f6b867fc640 10 monclient: _finish_hunting 0 2024-03-29T11:19:49.738 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:49.744+0000 7f6b867fc640 1 monclient: found mon.c 2024-03-29T11:19:49.738 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:49.744+0000 7f6b867fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:19:49.739 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:49.744+0000 7f6b867fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-29T11:19:49.739 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:49.744+0000 7f6b867fc640 1 -- 172.21.15.84:0/3076583940 --> v1:172.21.15.84:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b941cf150 con 0x7f6b9413f650 2024-03-29T11:19:49.739 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:49.744+0000 7f6b867fc640 10 monclient: _finish_auth 0 2024-03-29T11:19:49.739 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:49.744+0000 7f6b867fc640 10 monclient: _check_auth_tickets 2024-03-29T11:19:49.739 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:49.744+0000 7f6b867fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:19:49.739 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:49.745+0000 7f6b9bda5640 5 monclient: authenticate success, global_id 4247 2024-03-29T11:19:49.739 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:49.745+0000 7f6b9bda5640 10 monclient: _renew_subs 2024-03-29T11:19:49.739 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:49.745+0000 7f6b9bda5640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-29T11:19:49.739 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:49.745+0000 7f6b9bda5640 1 -- 172.21.15.84:0/3076583940 --> v1:172.21.15.84:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6b9413b830 con 0x7f6b9413f650 2024-03-29T11:19:49.739 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:49.745+0000 7f6b867fc640 1 -- 172.21.15.84:0/3076583940 <== mon.2 v1:172.21.15.84:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6b900038a0 con 0x7f6b9413f650 2024-03-29T11:19:49.739 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:49.745+0000 7f6b867fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:19:49.739 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:49.745+0000 7f6b867fc640 1 -- 172.21.15.84:0/3076583940 <== mon.2 v1:172.21.15.84:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4236868694 0 0) 0x7f6b90003c70 con 0x7f6b9413f650 2024-03-29T11:19:49.739 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:49.745+0000 7f6b867fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:49.739 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:49.745+0000 7f6b867fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:19:49.739 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:49.745+0000 7f6b867fc640 10 monclient: dump: 2024-03-29T11:19:49.740 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T11:19:49.740 INFO:teuthology.misc.health.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:19:49.740 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T11:18:53.864025+0000 2024-03-29T11:19:49.740 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T11:18:53.864025+0000 2024-03-29T11:19:49.740 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:49.740 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:19:49.740 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T11:19:49.740 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T11:19:49.740 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T11:19:49.740 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:19:49.740 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:49.745+0000 7f6b877fe640 4 set_mon_vals no callback set 2024-03-29T11:19:49.740 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:49.745+0000 7f6b9bda5640 10 client.4247.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:19:49.740 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:49.745+0000 7f6b9bda5640 10 monclient: _renew_subs 2024-03-29T11:19:49.740 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:49.745+0000 7f6b9bda5640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-29T11:19:49.740 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:49.745+0000 7f6b9bda5640 1 -- 172.21.15.84:0/3076583940 --> v1:172.21.15.84:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f6b941cb790 con 0x7f6b9413f650 2024-03-29T11:19:49.741 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:49.745+0000 7f6b9bda5640 1 librados: init done 2024-03-29T11:19:49.741 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:49.746+0000 7f6b9bda5640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:49.741 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:49.746+0000 7f6b9bda5640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:49.741 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:49.746+0000 7f6b9bda5640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-29T11:19:49.741 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:49.746+0000 7f6b9bda5640 1 -- 172.21.15.84:0/3076583940 --> v1:172.21.15.84:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6b9413bbd0 con 0x7f6b9413f650 2024-03-29T11:19:49.742 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:49.749+0000 7f6b867fc640 1 -- 172.21.15.84:0/3076583940 <== mon.2 v1:172.21.15.84:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4033942909 0 0) 0x7f6b90003dd0 con 0x7f6b9413f650 2024-03-29T11:19:49.742 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:49.750+0000 7f6b867fc640 1 -- 172.21.15.84:0/3076583940 <== mon.2 v1:172.21.15.84:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2808139587 0 0) 0x7f6b90084a80 con 0x7f6b9413f650 2024-03-29T11:19:49.742 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:49.750+0000 7f6b867fc640 10 client.4247.objecter ms_dispatch 0x7f6b94000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:19:49.742 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:49.750+0000 7f6b867fc640 3 client.4247.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:19:49.742 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:49.750+0000 7f6b867fc640 3 client.4247.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:19:49.742 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:49.750+0000 7f6b867fc640 20 client.4247.objecter dump_active .. 0 homeless 2024-03-29T11:19:49.743 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:49.750+0000 7f6b99b41640 10 monclient: get_auth_request con 0x7f6b7806e340 auth_method 0 2024-03-29T11:19:49.743 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:49.750+0000 7f6b867fc640 10 client.4247.objecter ms_handle_connect 0x7f6b7806e340 2024-03-29T11:19:49.743 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:49.757+0000 7f6b867fc640 1 -- 172.21.15.84:0/3076583940 <== mon.2 v1:172.21.15.84:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f6b90050360 con 0x7f6b9413f650 2024-03-29T11:19:49.743 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:49.757+0000 7f6b867fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:49.743 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:49.757+0000 7f6b867fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:19:50.099 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:50.122+0000 7f6b9bda5640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:19:50.099 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:50.122+0000 7f6b9bda5640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:19:50.099 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:50.122+0000 7f6b9bda5640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-29T11:19:50.099 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:50.122+0000 7f6b9bda5640 1 -- 172.21.15.84:0/3076583940 --> v1:172.21.15.84:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6b941ccfe0 con 0x7f6b9413f650 2024-03-29T11:19:50.100 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:50.123+0000 7f6b867fc640 1 -- 172.21.15.84:0/3076583940 <== mon.2 v1:172.21.15.84:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4209477549) 0x7f6b90001210 con 0x7f6b9413f650 2024-03-29T11:19:50.100 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:50.123+0000 7f6b867fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:19:50.100 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:50.123+0000 7f6b867fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:19:50.100 INFO:teuthology.misc.health.smithi084.stdout: 2024-03-29T11:19:50.100 INFO:teuthology.misc.health.smithi084.stdout:{"epoch":10,"fsid":"2c3e002f-9eb0-4860-853b-4ef1ef562213","created":"2024-03-29T11:19:32.029169+0000","modified":"2024-03-29T11:19:34.200299+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:34.200299+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"5c39e276-2034-4716-bbc0-57a634eefa92","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"cd580e99-109f-4bf1-88c0-a5987bc8b9ed","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"11e60a5a-051d-453f-a190-66387cb35592","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"2c329fbe-bbc4-4030-bf87-d711cad04679","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f4213d4a-bd36-4433-91ae-d7e2b02481d3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"f0cf44ea-6055-48af-805b-aed858102ff9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"d895eaa0-cc0b-4ee4-9b5a-2dfcd89a60b7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"9bc5a7bc-8f52-4db9-96d6-de5e68d52eaa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:19:50.104 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:50.126+0000 7f6b9bda5640 10 librados: watch_flush enter 2024-03-29T11:19:50.104 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:50.126+0000 7f6b9bda5640 10 librados: watch_flush exit 2024-03-29T11:19:50.104 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:50.126+0000 7f6b9bda5640 20 client.4247.objecter shutdown clearing up homeless session... 2024-03-29T11:19:50.104 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:50.126+0000 7f6b9bda5640 10 client.4247.objecter successfully canceled tick 2024-03-29T11:19:50.104 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:50.127+0000 7f6b9bda5640 1 -- 172.21.15.84:0/3076583940 >> v1:172.21.15.179:6816/2981134980 conn(0x7f6b7806e340 legacy=0x7f6b78070800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:50.104 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:50.127+0000 7f6b9bda5640 10 monclient: shutdown 2024-03-29T11:19:50.104 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:50.127+0000 7f6b9bda5640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:19:50.104 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:50.127+0000 7f6b9bda5640 1 -- 172.21.15.84:0/3076583940 >> v1:172.21.15.84:6790/0 conn(0x7f6b9413f650 legacy=0x7f6b941d7e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:50.104 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:50.127+0000 7f6b9bda5640 1 -- 172.21.15.84:0/3076583940 shutdown_connections 2024-03-29T11:19:50.104 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:50.127+0000 7f6b9bda5640 1 -- 172.21.15.84:0/3076583940 >> 172.21.15.84:0/3076583940 conn(0x7f6b94066a00 msgr2=0x7f6b9413ce40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:19:50.105 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:50.128+0000 7f6b9bda5640 1 -- 172.21.15.84:0/3076583940 shutdown_connections 2024-03-29T11:19:50.105 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:50.128+0000 7f6b9bda5640 1 -- 172.21.15.84:0/3076583940 wait complete. 2024-03-29T11:19:50.105 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:50.128+0000 7f6b9bda5640 1 librados: shutdown 2024-03-29T11:19:50.122 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:19:56.929 DEBUG:teuthology.orchestra.run.smithi084:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:19:57.068 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:57.090+0000 7fac7b0b3640 10 monclient: get_monmap_and_config 2024-03-29T11:19:57.068 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:57.090+0000 7fac7b0b3640 10 monclient: build_initial_monmap 2024-03-29T11:19:57.068 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:57.090+0000 7fac7b0b3640 10 monclient: monmap: 2024-03-29T11:19:57.068 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-29T11:19:57.068 INFO:teuthology.misc.health.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:19:57.068 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T11:19:57.092368+0000 2024-03-29T11:19:57.068 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T11:19:57.092368+0000 2024-03-29T11:19:57.068 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-29T11:19:57.068 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:19:57.068 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-29T11:19:57.069 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.noname-c 2024-03-29T11:19:57.069 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-b 2024-03-29T11:19:57.069 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:19:57.071 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:57.091+0000 7fac7b0b3640 1 Processor -- start 2024-03-29T11:19:57.071 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:57.091+0000 7fac7b0b3640 1 -- start start 2024-03-29T11:19:57.071 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:57.091+0000 7fac7b0b3640 10 monclient: init 2024-03-29T11:19:57.071 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:57.092+0000 7fac7b0b3640 10 monclient: _reopen_session rank -1 2024-03-29T11:19:57.071 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:57.092+0000 7fac7b0b3640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:19:57.071 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:57.092+0000 7fac7b0b3640 10 monclient(hunting): picked mon.noname-b con 0x7fac7406ade0 addr v1:172.21.15.84:6790/0 2024-03-29T11:19:57.071 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:57.092+0000 7fac7b0b3640 10 monclient(hunting): picked mon.noname-c con 0x7fac7406ed60 addr v1:172.21.15.179:6789/0 2024-03-29T11:19:57.072 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:57.092+0000 7fac7b0b3640 10 monclient(hunting): picked mon.noname-a con 0x7fac74072340 addr v1:172.21.15.84:6789/0 2024-03-29T11:19:57.072 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:57.092+0000 7fac7b0b3640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fac7406ff50 con 0x7fac74072340 2024-03-29T11:19:57.072 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:57.092+0000 7fac7b0b3640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fac74070180 con 0x7fac7406ed60 2024-03-29T11:19:57.072 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:57.092+0000 7fac7b0b3640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fac7406f310 con 0x7fac7406ade0 2024-03-29T11:19:57.072 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:57.092+0000 7fac7b0b3640 10 monclient(hunting): _renew_subs 2024-03-29T11:19:57.072 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:57.092+0000 7fac7b0b3640 10 monclient(hunting): authenticate will time out at 858.387787s 2024-03-29T11:19:57.072 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:57.092+0000 7fac73fff640 1 --1- >> v1:172.21.15.179:6789/0 conn(0x7fac7406ed60 0x7fac74071c30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.179:6789/0 says I am v1:172.21.15.84:52410/0 (socket says 172.21.15.84:52410) 2024-03-29T11:19:57.072 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:57.092+0000 7fac73fff640 1 -- 172.21.15.84:0/3380520743 learned_addr learned my addr 172.21.15.84:0/3380520743 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-29T11:19:57.072 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:57.093+0000 7fac737fe640 1 -- 172.21.15.84:0/3380520743 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 296755697 0 0) 0x7fac7406f310 con 0x7fac7406ade0 2024-03-29T11:19:57.072 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:57.093+0000 7fac737fe640 10 monclient(hunting): _negotiate 2024-03-29T11:19:57.072 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:57.093+0000 7fac737fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:57.072 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:57.093+0000 7fac737fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:57.072 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:57.093+0000 7fac737fe640 10 monclient(hunting): my global_id is 4250 2024-03-29T11:19:57.072 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:57.093+0000 7fac737fe640 1 -- 172.21.15.84:0/3380520743 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fac5c003520 con 0x7fac7406ade0 2024-03-29T11:19:57.073 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:57.093+0000 7fac737fe640 1 -- 172.21.15.84:0/3380520743 <== mon.2 v1:172.21.15.84:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3528673584 0 0) 0x7fac5c003520 con 0x7fac7406ade0 2024-03-29T11:19:57.073 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:57.093+0000 7fac737fe640 1 -- 172.21.15.84:0/3380520743 >> v1:172.21.15.179:6789/0 conn(0x7fac7406ed60 legacy=0x7fac74071c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:19:57.073 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:57.093+0000 7fac737fe640 1 -- 172.21.15.84:0/3380520743 >> v1:172.21.15.84:6789/0 conn(0x7fac74072340 legacy=0x7fac74151d40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:19:57.073 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:57.095+0000 7fac737fe640 10 monclient: _finish_hunting 0 2024-03-29T11:19:57.074 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:57.095+0000 7fac737fe640 1 monclient: found mon.noname-b 2024-03-29T11:19:57.074 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:57.095+0000 7fac737fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:19:57.074 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:57.095+0000 7fac737fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.84:6790/0 2024-03-29T11:19:57.074 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:57.095+0000 7fac737fe640 1 -- 172.21.15.84:0/3380520743 --> v1:172.21.15.84:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac740aeed0 con 0x7fac7406ade0 2024-03-29T11:19:57.074 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:57.095+0000 7fac737fe640 10 monclient: _finish_auth 0 2024-03-29T11:19:57.074 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:57.095+0000 7fac737fe640 10 monclient: _check_auth_tickets 2024-03-29T11:19:57.074 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:57.095+0000 7fac737fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:19:57.074 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:57.095+0000 7fac7b0b3640 5 monclient: authenticate success, global_id 4250 2024-03-29T11:19:57.076 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:57.095+0000 7fac7b0b3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:19:57.076 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:57.095+0000 7fac737fe640 1 -- 172.21.15.84:0/3380520743 <== mon.2 v1:172.21.15.84:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4236868694 0 0) 0x7fac64001690 con 0x7fac7406ade0 2024-03-29T11:19:57.076 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:57.095+0000 7fac737fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:57.076 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:57.095+0000 7fac737fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:19:57.076 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:57.095+0000 7fac737fe640 10 monclient: dump: 2024-03-29T11:19:57.076 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T11:19:57.076 INFO:teuthology.misc.health.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:19:57.076 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T11:18:53.864025+0000 2024-03-29T11:19:57.076 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T11:18:53.864025+0000 2024-03-29T11:19:57.076 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:57.076 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:19:57.076 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T11:19:57.076 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T11:19:57.076 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T11:19:57.077 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:19:57.077 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:57.095+0000 7fac7b0b3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:19:57.077 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:57.095+0000 7fac737fe640 1 -- 172.21.15.84:0/3380520743 <== mon.2 v1:172.21.15.84:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fac64001860 con 0x7fac7406ade0 2024-03-29T11:19:57.077 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:57.095+0000 7fac737fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:19:57.077 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:57.095+0000 7fac7b0b3640 10 monclient: get_monmap_and_config success 2024-03-29T11:19:57.077 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:57.095+0000 7fac7b0b3640 4 set_mon_vals no callback set 2024-03-29T11:19:57.077 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:57.096+0000 7fac737fe640 1 -- 172.21.15.84:0/3380520743 <== mon.2 v1:172.21.15.84:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4236868694 0 0) 0x7fac64001cb0 con 0x7fac7406ade0 2024-03-29T11:19:57.077 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:57.096+0000 7fac737fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:57.077 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:57.096+0000 7fac737fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:19:57.077 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:57.096+0000 7fac737fe640 10 monclient: dump: 2024-03-29T11:19:57.077 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T11:19:57.077 INFO:teuthology.misc.health.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:19:57.077 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T11:18:53.864025+0000 2024-03-29T11:19:57.077 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T11:18:53.864025+0000 2024-03-29T11:19:57.077 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:57.077 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:19:57.077 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T11:19:57.078 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T11:19:57.078 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T11:19:57.078 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:19:57.078 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:57.096+0000 7fac7b0b3640 10 monclient: shutdown 2024-03-29T11:19:57.078 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:57.096+0000 7fac7b0b3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:19:57.078 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:57.096+0000 7fac7b0b3640 1 -- 172.21.15.84:0/3380520743 >> v1:172.21.15.84:6790/0 conn(0x7fac7406ade0 legacy=0x7fac741499e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:57.078 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:57.096+0000 7fac7b0b3640 1 -- 172.21.15.84:0/3380520743 shutdown_connections 2024-03-29T11:19:57.078 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:57.096+0000 7fac7b0b3640 1 -- 172.21.15.84:0/3380520743 >> 172.21.15.84:0/3380520743 conn(0x7fac740b4930 msgr2=0x7fac740b4d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:19:57.083 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:57.096+0000 7fac7b0b3640 1 -- 172.21.15.84:0/3380520743 shutdown_connections 2024-03-29T11:19:57.083 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:57.096+0000 7fac7b0b3640 1 -- 172.21.15.84:0/3380520743 wait complete. 2024-03-29T11:19:57.084 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:57.097+0000 7fac7b0b3640 10 monclient: build_initial_monmap 2024-03-29T11:19:57.084 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:57.097+0000 7fac7b0b3640 10 monclient: monmap: 2024-03-29T11:19:57.084 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-29T11:19:57.084 INFO:teuthology.misc.health.smithi084.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:19:57.084 INFO:teuthology.misc.health.smithi084.stderr:last_changed 0.000000 2024-03-29T11:19:57.084 INFO:teuthology.misc.health.smithi084.stderr:created 0.000000 2024-03-29T11:19:57.084 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-29T11:19:57.084 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:19:57.084 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-29T11:19:57.084 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.noname-b 2024-03-29T11:19:57.084 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-c 2024-03-29T11:19:57.084 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:19:57.084 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:57.098+0000 7fac7b0b3640 1 librados: starting msgr at 2024-03-29T11:19:57.084 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:57.098+0000 7fac7b0b3640 1 librados: starting objecter 2024-03-29T11:19:57.085 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:57.098+0000 7fac7b0b3640 1 Processor -- start 2024-03-29T11:19:57.085 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:57.098+0000 7fac7b0b3640 1 -- start start 2024-03-29T11:19:57.085 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:57.098+0000 7fac7b0b3640 1 librados: setting wanted keys 2024-03-29T11:19:57.085 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:57.098+0000 7fac7b0b3640 1 librados: calling monclient init 2024-03-29T11:19:57.085 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:57.098+0000 7fac7b0b3640 10 monclient: init 2024-03-29T11:19:57.085 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:57.099+0000 7fac7b0b3640 10 monclient: _reopen_session rank -1 2024-03-29T11:19:57.085 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:57.099+0000 7fac7b0b3640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:19:57.085 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:57.100+0000 7fac7b0b3640 10 monclient(hunting): picked mon.noname-a con 0x7fac7406ade0 addr v1:172.21.15.84:6789/0 2024-03-29T11:19:57.085 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:57.100+0000 7fac7b0b3640 10 monclient(hunting): picked mon.noname-b con 0x7fac7406ed60 addr v1:172.21.15.179:6789/0 2024-03-29T11:19:57.085 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:57.100+0000 7fac7b0b3640 10 monclient(hunting): picked mon.noname-c con 0x7fac74072340 addr v1:172.21.15.84:6790/0 2024-03-29T11:19:57.085 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:57.100+0000 7fac7b0b3640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fac740ab8e0 con 0x7fac7406ade0 2024-03-29T11:19:57.085 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:57.100+0000 7fac7b0b3640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fac741df270 con 0x7fac7406ed60 2024-03-29T11:19:57.085 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:57.100+0000 7fac7b0b3640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fac741def90 con 0x7fac74072340 2024-03-29T11:19:57.085 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:57.100+0000 7fac7b0b3640 10 monclient(hunting): _renew_subs 2024-03-29T11:19:57.085 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:57.100+0000 7fac7b0b3640 10 monclient(hunting): authenticate will time out at 858.395338s 2024-03-29T11:19:57.086 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:57.100+0000 7fac79650640 1 --1- >> v1:172.21.15.84:6790/0 conn(0x7fac74072340 0x7fac740a9c80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.84:6790/0 says I am v1:172.21.15.84:49848/0 (socket says 172.21.15.84:49848) 2024-03-29T11:19:57.086 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:57.100+0000 7fac79650640 1 -- 172.21.15.84:0/2729741906 learned_addr learned my addr 172.21.15.84:0/2729741906 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-29T11:19:57.086 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:57.100+0000 7fac717fa640 10 client.?.objecter ms_handle_connect 0x7fac74072340 2024-03-29T11:19:57.086 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:57.100+0000 7fac717fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:19:57.086 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:57.100+0000 7fac717fa640 1 -- 172.21.15.84:0/2729741906 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2528782424 0 0) 0x7fac741def90 con 0x7fac74072340 2024-03-29T11:19:57.086 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:57.100+0000 7fac717fa640 10 monclient(hunting): _negotiate 2024-03-29T11:19:57.086 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:57.100+0000 7fac717fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:57.086 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:57.100+0000 7fac717fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:57.086 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:57.100+0000 7fac717fa640 10 monclient(hunting): my global_id is 4253 2024-03-29T11:19:57.086 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:57.101+0000 7fac717fa640 1 -- 172.21.15.84:0/2729741906 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fac60004eb0 con 0x7fac74072340 2024-03-29T11:19:57.086 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:57.101+0000 7fac717fa640 1 -- 172.21.15.84:0/2729741906 <== mon.2 v1:172.21.15.84:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3142163398 0 0) 0x7fac60004eb0 con 0x7fac74072340 2024-03-29T11:19:57.086 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:57.101+0000 7fac717fa640 1 -- 172.21.15.84:0/2729741906 --> v1:172.21.15.84:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fac741def90 con 0x7fac74072340 2024-03-29T11:19:57.086 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:57.101+0000 7fac717fa640 1 -- 172.21.15.84:0/2729741906 <== mon.2 v1:172.21.15.84:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4236868694 0 0) 0x7fac6c001ad0 con 0x7fac74072340 2024-03-29T11:19:57.086 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:57.101+0000 7fac717fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:57.087 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:57.101+0000 7fac717fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:19:57.087 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:57.101+0000 7fac717fa640 10 monclient(hunting): dump: 2024-03-29T11:19:57.087 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T11:19:57.087 INFO:teuthology.misc.health.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:19:57.087 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T11:18:53.864025+0000 2024-03-29T11:19:57.087 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T11:18:53.864025+0000 2024-03-29T11:19:57.087 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:57.087 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:19:57.087 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T11:19:57.087 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T11:19:57.087 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T11:19:57.087 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:19:57.087 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:57.101+0000 7fac717fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:19:57.087 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:57.101+0000 7fac717fa640 1 -- 172.21.15.84:0/2729741906 <== mon.2 v1:172.21.15.84:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2560082595 0 0) 0x7fac741def90 con 0x7fac74072340 2024-03-29T11:19:57.087 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:57.101+0000 7fac717fa640 1 -- 172.21.15.84:0/2729741906 >> v1:172.21.15.179:6789/0 conn(0x7fac7406ed60 legacy=0x7fac740a9570 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:19:57.088 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:57.101+0000 7fac717fa640 1 -- 172.21.15.84:0/2729741906 >> v1:172.21.15.84:6789/0 conn(0x7fac7406ade0 legacy=0x7fac741de880 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:19:57.088 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:57.103+0000 7fac717fa640 10 monclient: _finish_hunting 0 2024-03-29T11:19:57.088 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:57.103+0000 7fac717fa640 1 monclient: found mon.c 2024-03-29T11:19:57.088 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:57.103+0000 7fac717fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:19:57.088 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:57.103+0000 7fac717fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-29T11:19:57.088 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:57.103+0000 7fac717fa640 1 -- 172.21.15.84:0/2729741906 --> v1:172.21.15.84:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac740ae4a0 con 0x7fac74072340 2024-03-29T11:19:57.088 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:57.103+0000 7fac717fa640 10 monclient: _finish_auth 0 2024-03-29T11:19:57.088 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:57.103+0000 7fac717fa640 10 monclient: _check_auth_tickets 2024-03-29T11:19:57.088 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:57.103+0000 7fac717fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:19:57.088 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:57.103+0000 7fac7b0b3640 5 monclient: authenticate success, global_id 4253 2024-03-29T11:19:57.088 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:57.103+0000 7fac7b0b3640 10 monclient: _renew_subs 2024-03-29T11:19:57.088 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:57.103+0000 7fac7b0b3640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-29T11:19:57.088 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:57.103+0000 7fac7b0b3640 1 -- 172.21.15.84:0/2729741906 --> v1:172.21.15.84:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fac740ae6f0 con 0x7fac74072340 2024-03-29T11:19:57.088 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:57.103+0000 7fac7b0b3640 10 client.4253.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:19:57.089 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:57.103+0000 7fac7b0b3640 10 monclient: _renew_subs 2024-03-29T11:19:57.089 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:57.103+0000 7fac7b0b3640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-29T11:19:57.089 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:57.103+0000 7fac7b0b3640 1 -- 172.21.15.84:0/2729741906 --> v1:172.21.15.84:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fac741e6d40 con 0x7fac74072340 2024-03-29T11:19:57.089 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:57.103+0000 7fac7b0b3640 1 librados: init done 2024-03-29T11:19:57.089 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:57.103+0000 7fac717fa640 1 -- 172.21.15.84:0/2729741906 <== mon.2 v1:172.21.15.84:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fac6c0020a0 con 0x7fac74072340 2024-03-29T11:19:57.089 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:57.103+0000 7fac717fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:19:57.089 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:57.103+0000 7fac72ffd640 4 set_mon_vals no callback set 2024-03-29T11:19:57.089 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:57.104+0000 7fac717fa640 1 -- 172.21.15.84:0/2729741906 <== mon.2 v1:172.21.15.84:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4236868694 0 0) 0x7fac6c004d80 con 0x7fac74072340 2024-03-29T11:19:57.089 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:57.104+0000 7fac717fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:57.089 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:57.104+0000 7fac717fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:19:57.089 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:57.104+0000 7fac717fa640 10 monclient: dump: 2024-03-29T11:19:57.089 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T11:19:57.089 INFO:teuthology.misc.health.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:19:57.089 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T11:18:53.864025+0000 2024-03-29T11:19:57.090 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T11:18:53.864025+0000 2024-03-29T11:19:57.090 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:57.090 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:19:57.090 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T11:19:57.090 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T11:19:57.090 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T11:19:57.090 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:19:57.090 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:57.105+0000 7fac7b0b3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:57.090 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:57.105+0000 7fac7b0b3640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:57.090 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:57.105+0000 7fac7b0b3640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-29T11:19:57.090 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:57.105+0000 7fac7b0b3640 1 -- 172.21.15.84:0/2729741906 --> v1:172.21.15.84:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fac741df450 con 0x7fac74072340 2024-03-29T11:19:57.091 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:57.107+0000 7fac717fa640 1 -- 172.21.15.84:0/2729741906 <== mon.2 v1:172.21.15.84:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4033942909 0 0) 0x7fac6c01add0 con 0x7fac74072340 2024-03-29T11:19:57.091 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:57.107+0000 7fac717fa640 1 -- 172.21.15.84:0/2729741906 <== mon.2 v1:172.21.15.84:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2808139587 0 0) 0x7fac6c084a00 con 0x7fac74072340 2024-03-29T11:19:57.091 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:57.107+0000 7fac717fa640 10 client.4253.objecter ms_dispatch 0x7fac74000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:19:57.092 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:57.107+0000 7fac717fa640 3 client.4253.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:19:57.092 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:57.107+0000 7fac717fa640 3 client.4253.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:19:57.092 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:57.107+0000 7fac717fa640 20 client.4253.objecter dump_active .. 0 homeless 2024-03-29T11:19:57.092 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:57.108+0000 7fac78e4f640 10 monclient: get_auth_request con 0x7fac6006dcd0 auth_method 0 2024-03-29T11:19:57.092 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:57.108+0000 7fac717fa640 10 client.4253.objecter ms_handle_connect 0x7fac6006dcd0 2024-03-29T11:19:57.092 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:57.113+0000 7fac717fa640 1 -- 172.21.15.84:0/2729741906 <== mon.2 v1:172.21.15.84:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fac6c0502e0 con 0x7fac74072340 2024-03-29T11:19:57.092 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:57.113+0000 7fac717fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:57.092 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:57.113+0000 7fac717fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:19:57.408 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:57.431+0000 7fac7b0b3640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:19:57.408 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:57.431+0000 7fac7b0b3640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:19:57.408 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:57.431+0000 7fac7b0b3640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-29T11:19:57.408 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:57.431+0000 7fac7b0b3640 1 -- 172.21.15.84:0/2729741906 --> v1:172.21.15.84:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fac74075f10 con 0x7fac74072340 2024-03-29T11:19:57.409 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:57.432+0000 7fac717fa640 1 -- 172.21.15.84:0/2729741906 <== mon.2 v1:172.21.15.84:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4209477549) 0x7fac6c0502e0 con 0x7fac74072340 2024-03-29T11:19:57.409 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:57.432+0000 7fac717fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:19:57.409 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:57.432+0000 7fac717fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:19:57.409 INFO:teuthology.misc.health.smithi084.stdout: 2024-03-29T11:19:57.409 INFO:teuthology.misc.health.smithi084.stdout:{"epoch":10,"fsid":"2c3e002f-9eb0-4860-853b-4ef1ef562213","created":"2024-03-29T11:19:32.029169+0000","modified":"2024-03-29T11:19:34.200299+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:34.200299+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"5c39e276-2034-4716-bbc0-57a634eefa92","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"cd580e99-109f-4bf1-88c0-a5987bc8b9ed","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"11e60a5a-051d-453f-a190-66387cb35592","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"2c329fbe-bbc4-4030-bf87-d711cad04679","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f4213d4a-bd36-4433-91ae-d7e2b02481d3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"f0cf44ea-6055-48af-805b-aed858102ff9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"d895eaa0-cc0b-4ee4-9b5a-2dfcd89a60b7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"9bc5a7bc-8f52-4db9-96d6-de5e68d52eaa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:19:57.412 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:57.435+0000 7fac7b0b3640 10 librados: watch_flush enter 2024-03-29T11:19:57.412 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:57.435+0000 7fac7b0b3640 10 librados: watch_flush exit 2024-03-29T11:19:57.412 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:57.435+0000 7fac7b0b3640 20 client.4253.objecter shutdown clearing up homeless session... 2024-03-29T11:19:57.412 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:57.435+0000 7fac7b0b3640 10 client.4253.objecter successfully canceled tick 2024-03-29T11:19:57.412 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:57.435+0000 7fac7b0b3640 1 -- 172.21.15.84:0/2729741906 >> v1:172.21.15.179:6816/2981134980 conn(0x7fac6006dcd0 legacy=0x7fac60070190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:57.412 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:57.435+0000 7fac7b0b3640 10 monclient: shutdown 2024-03-29T11:19:57.413 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:57.435+0000 7fac7b0b3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:19:57.413 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:57.435+0000 7fac7b0b3640 1 -- 172.21.15.84:0/2729741906 >> v1:172.21.15.84:6790/0 conn(0x7fac74072340 legacy=0x7fac740a9c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:57.413 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:57.435+0000 7fac7b0b3640 1 -- 172.21.15.84:0/2729741906 shutdown_connections 2024-03-29T11:19:57.413 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:57.435+0000 7fac7b0b3640 1 -- 172.21.15.84:0/2729741906 >> 172.21.15.84:0/2729741906 conn(0x7fac740b4930 msgr2=0x7fac740afa80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:19:57.413 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:57.435+0000 7fac7b0b3640 1 -- 172.21.15.84:0/2729741906 shutdown_connections 2024-03-29T11:19:57.413 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:57.436+0000 7fac7b0b3640 1 -- 172.21.15.84:0/2729741906 wait complete. 2024-03-29T11:19:57.413 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:19:57.436+0000 7fac7b0b3640 1 librados: shutdown 2024-03-29T11:19:57.426 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:20:04.233 DEBUG:teuthology.orchestra.run.smithi084:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:20:04.392 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:04.415+0000 7fa303fff640 10 monclient: get_monmap_and_config 2024-03-29T11:20:04.392 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:04.415+0000 7fa303fff640 10 monclient: build_initial_monmap 2024-03-29T11:20:04.392 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:04.415+0000 7fa303fff640 10 monclient: monmap: 2024-03-29T11:20:04.392 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-29T11:20:04.392 INFO:teuthology.misc.health.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:20:04.392 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T11:20:04.416258+0000 2024-03-29T11:20:04.392 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T11:20:04.416258+0000 2024-03-29T11:20:04.392 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-29T11:20:04.393 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:20:04.393 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-29T11:20:04.393 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.noname-c 2024-03-29T11:20:04.393 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-b 2024-03-29T11:20:04.393 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:20:04.393 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:04.416+0000 7fa303fff640 1 Processor -- start 2024-03-29T11:20:04.394 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:04.416+0000 7fa303fff640 1 -- start start 2024-03-29T11:20:04.394 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:04.416+0000 7fa303fff640 10 monclient: init 2024-03-29T11:20:04.394 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:04.417+0000 7fa303fff640 10 monclient: _reopen_session rank -1 2024-03-29T11:20:04.394 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:04.417+0000 7fa303fff640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:20:04.397 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:04.419+0000 7fa302ffd640 1 --1- >> v1:172.21.15.84:6790/0 conn(0x7fa2fc0389e0 0x7fa2fc033520 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.84:6790/0 says I am v1:172.21.15.84:46751/0 (socket says 172.21.15.84:46751) 2024-03-29T11:20:04.398 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:04.420+0000 7fa303fff640 10 monclient(hunting): picked mon.noname-b con 0x7fa2fc0389e0 addr v1:172.21.15.84:6790/0 2024-03-29T11:20:04.398 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:04.420+0000 7fa303fff640 10 monclient(hunting): picked mon.noname-a con 0x7fa2fc038dc0 addr v1:172.21.15.84:6789/0 2024-03-29T11:20:04.398 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:04.420+0000 7fa303fff640 10 monclient(hunting): picked mon.noname-c con 0x7fa2fc030660 addr v1:172.21.15.179:6789/0 2024-03-29T11:20:04.398 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:04.420+0000 7fa303fff640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa2fc0c54c0 con 0x7fa2fc038dc0 2024-03-29T11:20:04.398 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:04.420+0000 7fa302ffd640 1 -- 172.21.15.84:0/2814239347 learned_addr learned my addr 172.21.15.84:0/2814239347 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-29T11:20:04.398 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:04.420+0000 7fa303fff640 1 -- 172.21.15.84:0/2814239347 --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa2fc036430 con 0x7fa2fc030660 2024-03-29T11:20:04.398 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:04.420+0000 7fa303fff640 1 -- 172.21.15.84:0/2814239347 --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa2fc039840 con 0x7fa2fc0389e0 2024-03-29T11:20:04.398 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:04.420+0000 7fa303fff640 10 monclient(hunting): _renew_subs 2024-03-29T11:20:04.398 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:04.420+0000 7fa303fff640 10 monclient(hunting): authenticate will time out at 865.714909s 2024-03-29T11:20:04.398 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:04.420+0000 7fa301ffb640 1 -- 172.21.15.84:0/2814239347 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2735392858 0 0) 0x7fa2fc039840 con 0x7fa2fc0389e0 2024-03-29T11:20:04.398 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:04.421+0000 7fa301ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:20:04.398 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:04.421+0000 7fa301ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:04.398 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:04.421+0000 7fa301ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:04.398 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:04.421+0000 7fa301ffb640 10 monclient(hunting): my global_id is 4256 2024-03-29T11:20:04.399 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:04.421+0000 7fa301ffb640 1 -- 172.21.15.84:0/2814239347 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa2ec003520 con 0x7fa2fc0389e0 2024-03-29T11:20:04.399 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:04.421+0000 7fa301ffb640 1 -- 172.21.15.84:0/2814239347 <== mon.2 v1:172.21.15.84:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1359333210 0 0) 0x7fa2ec003520 con 0x7fa2fc0389e0 2024-03-29T11:20:04.399 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:04.421+0000 7fa301ffb640 1 -- 172.21.15.84:0/2814239347 >> v1:172.21.15.179:6789/0 conn(0x7fa2fc030660 legacy=0x7fa2fc0d3170 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:20:04.399 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:04.423+0000 7fa301ffb640 1 -- 172.21.15.84:0/2814239347 >> v1:172.21.15.84:6789/0 conn(0x7fa2fc038dc0 legacy=0x7fa2fc0cadd0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:20:04.401 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:04.423+0000 7fa301ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:20:04.401 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:04.423+0000 7fa301ffb640 1 monclient: found mon.noname-b 2024-03-29T11:20:04.401 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:04.423+0000 7fa301ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:20:04.401 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:04.423+0000 7fa301ffb640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.84:6790/0 2024-03-29T11:20:04.401 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:04.423+0000 7fa301ffb640 1 -- 172.21.15.84:0/2814239347 --> v1:172.21.15.84:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2fc035690 con 0x7fa2fc0389e0 2024-03-29T11:20:04.401 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:04.423+0000 7fa301ffb640 10 monclient: _finish_auth 0 2024-03-29T11:20:04.401 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:04.423+0000 7fa301ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:20:04.401 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:04.423+0000 7fa301ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:20:04.401 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:04.423+0000 7fa301ffb640 1 -- 172.21.15.84:0/2814239347 <== mon.2 v1:172.21.15.84:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4236868694 0 0) 0x7fa2f4001690 con 0x7fa2fc0389e0 2024-03-29T11:20:04.401 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:04.423+0000 7fa301ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:04.401 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:04.423+0000 7fa301ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:20:04.401 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:04.423+0000 7fa301ffb640 10 monclient: dump: 2024-03-29T11:20:04.401 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T11:20:04.401 INFO:teuthology.misc.health.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:20:04.402 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T11:18:53.864025+0000 2024-03-29T11:20:04.402 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T11:18:53.864025+0000 2024-03-29T11:20:04.402 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:04.402 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:20:04.402 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T11:20:04.402 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T11:20:04.402 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T11:20:04.402 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:20:04.403 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:04.425+0000 7fa303fff640 5 monclient: authenticate success, global_id 4256 2024-03-29T11:20:04.403 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:04.425+0000 7fa303fff640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:20:04.403 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:04.426+0000 7fa301ffb640 1 -- 172.21.15.84:0/2814239347 <== mon.2 v1:172.21.15.84:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa2f4001920 con 0x7fa2fc0389e0 2024-03-29T11:20:04.403 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:04.426+0000 7fa301ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:20:04.403 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:04.426+0000 7fa301ffb640 1 -- 172.21.15.84:0/2814239347 <== mon.2 v1:172.21.15.84:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4236868694 0 0) 0x7fa2f4001cb0 con 0x7fa2fc0389e0 2024-03-29T11:20:04.405 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:04.426+0000 7fa303fff640 10 monclient: get_monmap_and_config success 2024-03-29T11:20:04.405 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:04.426+0000 7fa303fff640 4 set_mon_vals no callback set 2024-03-29T11:20:04.405 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:04.427+0000 7fa301ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:04.405 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:04.427+0000 7fa301ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:20:04.405 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:04.427+0000 7fa301ffb640 10 monclient: dump: 2024-03-29T11:20:04.405 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T11:20:04.405 INFO:teuthology.misc.health.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:20:04.405 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T11:18:53.864025+0000 2024-03-29T11:20:04.405 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T11:18:53.864025+0000 2024-03-29T11:20:04.405 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:04.405 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:20:04.406 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T11:20:04.406 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T11:20:04.406 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T11:20:04.406 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:20:04.406 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:04.427+0000 7fa303fff640 10 monclient: shutdown 2024-03-29T11:20:04.406 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:04.427+0000 7fa303fff640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:20:04.406 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:04.427+0000 7fa303fff640 1 -- 172.21.15.84:0/2814239347 >> v1:172.21.15.84:6790/0 conn(0x7fa2fc0389e0 legacy=0x7fa2fc033520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:04.406 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:04.428+0000 7fa303fff640 1 -- 172.21.15.84:0/2814239347 shutdown_connections 2024-03-29T11:20:04.406 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:04.428+0000 7fa303fff640 1 -- 172.21.15.84:0/2814239347 >> 172.21.15.84:0/2814239347 conn(0x7fa2fc005c00 msgr2=0x7fa2fc006010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:20:04.409 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:04.433+0000 7fa303fff640 1 -- 172.21.15.84:0/2814239347 shutdown_connections 2024-03-29T11:20:04.409 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:04.433+0000 7fa303fff640 1 -- 172.21.15.84:0/2814239347 wait complete. 2024-03-29T11:20:04.410 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:04.434+0000 7fa303fff640 10 monclient: build_initial_monmap 2024-03-29T11:20:04.410 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:04.434+0000 7fa303fff640 10 monclient: monmap: 2024-03-29T11:20:04.410 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-29T11:20:04.410 INFO:teuthology.misc.health.smithi084.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:20:04.410 INFO:teuthology.misc.health.smithi084.stderr:last_changed 0.000000 2024-03-29T11:20:04.410 INFO:teuthology.misc.health.smithi084.stderr:created 0.000000 2024-03-29T11:20:04.411 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-29T11:20:04.411 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:20:04.411 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-29T11:20:04.411 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.noname-b 2024-03-29T11:20:04.411 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-c 2024-03-29T11:20:04.411 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:20:04.412 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:04.434+0000 7fa303fff640 1 librados: starting msgr at 2024-03-29T11:20:04.412 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:04.434+0000 7fa303fff640 1 librados: starting objecter 2024-03-29T11:20:04.412 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:04.434+0000 7fa303fff640 1 Processor -- start 2024-03-29T11:20:04.412 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:04.435+0000 7fa303fff640 1 -- start start 2024-03-29T11:20:04.412 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:04.435+0000 7fa303fff640 1 librados: setting wanted keys 2024-03-29T11:20:04.412 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:04.435+0000 7fa303fff640 1 librados: calling monclient init 2024-03-29T11:20:04.412 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:04.435+0000 7fa303fff640 10 monclient: init 2024-03-29T11:20:04.413 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:04.436+0000 7fa303fff640 10 monclient: _reopen_session rank -1 2024-03-29T11:20:04.413 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:04.436+0000 7fa303fff640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:20:04.416 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:04.439+0000 7fa303fff640 10 monclient(hunting): picked mon.noname-c con 0x7fa2fc030660 addr v1:172.21.15.84:6790/0 2024-03-29T11:20:04.417 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:04.439+0000 7fa303fff640 10 monclient(hunting): picked mon.noname-b con 0x7fa2fc0389e0 addr v1:172.21.15.179:6789/0 2024-03-29T11:20:04.417 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:04.439+0000 7fa302ffd640 1 --1- >> v1:172.21.15.84:6790/0 conn(0x7fa2fc030660 0x7fa2fc15c900 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.84:6790/0 says I am v1:172.21.15.84:46859/0 (socket says 172.21.15.84:46859) 2024-03-29T11:20:04.417 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:04.439+0000 7fa303fff640 10 monclient(hunting): picked mon.noname-a con 0x7fa2fc15d010 addr v1:172.21.15.84:6789/0 2024-03-29T11:20:04.417 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:04.439+0000 7fa302ffd640 1 -- 172.21.15.84:0/2126530619 learned_addr learned my addr 172.21.15.84:0/2126530619 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-29T11:20:04.417 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:04.439+0000 7fa303fff640 1 -- 172.21.15.84:0/2126530619 --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa2fc0a55d0 con 0x7fa2fc15d010 2024-03-29T11:20:04.417 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:04.439+0000 7fa303fff640 1 -- 172.21.15.84:0/2126530619 --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa2fc02e9c0 con 0x7fa2fc0389e0 2024-03-29T11:20:04.417 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:04.439+0000 7fa303fff640 1 -- 172.21.15.84:0/2126530619 --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa2fc02eee0 con 0x7fa2fc030660 2024-03-29T11:20:04.417 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:04.439+0000 7fa303fff640 10 monclient(hunting): _renew_subs 2024-03-29T11:20:04.417 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:04.439+0000 7fa303fff640 10 monclient(hunting): authenticate will time out at 865.733763s 2024-03-29T11:20:04.417 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:04.440+0000 7fa2ebfff640 10 client.?.objecter ms_handle_connect 0x7fa2fc030660 2024-03-29T11:20:04.417 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:04.440+0000 7fa2ebfff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:20:04.424 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:04.443+0000 7fa2ebfff640 1 -- 172.21.15.84:0/2126530619 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2918268073 0 0) 0x7fa2fc02eee0 con 0x7fa2fc030660 2024-03-29T11:20:04.424 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:04.443+0000 7fa2ebfff640 10 monclient(hunting): _negotiate 2024-03-29T11:20:04.424 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:04.443+0000 7fa2ebfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:04.424 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:04.443+0000 7fa2ebfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:04.424 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:04.443+0000 7fa2ebfff640 10 monclient(hunting): my global_id is 4259 2024-03-29T11:20:04.424 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:04.443+0000 7fa2ebfff640 1 -- 172.21.15.84:0/2126530619 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa2f0004eb0 con 0x7fa2fc030660 2024-03-29T11:20:04.424 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:04.443+0000 7fa2ebfff640 10 client.?.objecter ms_handle_connect 0x7fa2fc15d010 2024-03-29T11:20:04.425 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:04.443+0000 7fa2ebfff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:20:04.425 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:04.443+0000 7fa2ebfff640 1 -- 172.21.15.84:0/2126530619 <== mon.0 v1:172.21.15.84:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2699343422 0 0) 0x7fa2fc0a55d0 con 0x7fa2fc15d010 2024-03-29T11:20:04.425 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:04.444+0000 7fa2ebfff640 10 monclient(hunting): _negotiate 2024-03-29T11:20:04.425 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:04.444+0000 7fa2ebfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:04.425 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:04.444+0000 7fa2ebfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:04.425 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:04.444+0000 7fa2ebfff640 10 monclient(hunting): my global_id is 4254 2024-03-29T11:20:04.425 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:04.444+0000 7fa2ebfff640 1 -- 172.21.15.84:0/2126530619 --> v1:172.21.15.84:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa2fc02eee0 con 0x7fa2fc15d010 2024-03-29T11:20:04.425 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:04.444+0000 7fa2ebfff640 1 -- 172.21.15.84:0/2126530619 <== mon.2 v1:172.21.15.84:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3944806229 0 0) 0x7fa2f0004eb0 con 0x7fa2fc030660 2024-03-29T11:20:04.425 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:04.444+0000 7fa2ebfff640 1 -- 172.21.15.84:0/2126530619 --> v1:172.21.15.84:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa2fc0a55d0 con 0x7fa2fc030660 2024-03-29T11:20:04.425 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:04.444+0000 7fa2ebfff640 1 -- 172.21.15.84:0/2126530619 <== mon.2 v1:172.21.15.84:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4236868694 0 0) 0x7fa2f4001610 con 0x7fa2fc030660 2024-03-29T11:20:04.425 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:04.444+0000 7fa2ebfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:04.425 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:04.444+0000 7fa2ebfff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:20:04.425 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:04.444+0000 7fa2ebfff640 10 monclient(hunting): dump: 2024-03-29T11:20:04.425 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T11:20:04.426 INFO:teuthology.misc.health.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:20:04.426 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T11:18:53.864025+0000 2024-03-29T11:20:04.426 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T11:18:53.864025+0000 2024-03-29T11:20:04.426 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:04.426 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:20:04.426 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T11:20:04.426 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T11:20:04.426 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T11:20:04.426 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:20:04.426 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:04.444+0000 7fa2ebfff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:20:04.426 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:04.444+0000 7fa2ebfff640 1 -- 172.21.15.84:0/2126530619 <== mon.2 v1:172.21.15.84:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 141865571 0 0) 0x7fa2fc0a55d0 con 0x7fa2fc030660 2024-03-29T11:20:04.426 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:04.444+0000 7fa2ebfff640 1 -- 172.21.15.84:0/2126530619 >> v1:172.21.15.179:6789/0 conn(0x7fa2fc0389e0 legacy=0x7fa2fc0c2070 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:20:04.426 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:04.445+0000 7fa2ebfff640 1 -- 172.21.15.84:0/2126530619 >> v1:172.21.15.84:6789/0 conn(0x7fa2fc15d010 legacy=0x7fa2fc168180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:04.426 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:04.445+0000 7fa2ebfff640 10 monclient: _finish_hunting 0 2024-03-29T11:20:04.427 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:04.445+0000 7fa2ebfff640 1 monclient: found mon.c 2024-03-29T11:20:04.427 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:04.445+0000 7fa2ebfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:20:04.427 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:04.445+0000 7fa2ebfff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-29T11:20:04.427 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:04.446+0000 7fa2ebfff640 1 -- 172.21.15.84:0/2126530619 --> v1:172.21.15.84:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2fc02f070 con 0x7fa2fc030660 2024-03-29T11:20:04.427 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:04.446+0000 7fa2ebfff640 10 monclient: _finish_auth 0 2024-03-29T11:20:04.427 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:04.446+0000 7fa2ebfff640 10 monclient: _check_auth_tickets 2024-03-29T11:20:04.427 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:04.446+0000 7fa2ebfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:20:04.427 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:04.446+0000 7fa303fff640 5 monclient: authenticate success, global_id 4259 2024-03-29T11:20:04.427 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:04.446+0000 7fa303fff640 10 monclient: _renew_subs 2024-03-29T11:20:04.427 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:04.446+0000 7fa303fff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-29T11:20:04.427 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:04.446+0000 7fa303fff640 1 -- 172.21.15.84:0/2126530619 --> v1:172.21.15.84:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa2fc02e600 con 0x7fa2fc030660 2024-03-29T11:20:04.427 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:04.446+0000 7fa303fff640 10 client.4259.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:20:04.427 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:04.446+0000 7fa303fff640 10 monclient: _renew_subs 2024-03-29T11:20:04.427 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:04.446+0000 7fa303fff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-29T11:20:04.427 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:04.446+0000 7fa303fff640 1 -- 172.21.15.84:0/2126530619 --> v1:172.21.15.84:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa2fc0c7740 con 0x7fa2fc030660 2024-03-29T11:20:04.428 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:04.446+0000 7fa303fff640 1 librados: init done 2024-03-29T11:20:04.428 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:04.446+0000 7fa2ebfff640 1 -- 172.21.15.84:0/2126530619 <== mon.2 v1:172.21.15.84:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa2f4004bf0 con 0x7fa2fc030660 2024-03-29T11:20:04.428 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:04.446+0000 7fa2ebfff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:20:04.428 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:04.446+0000 7fa2ebfff640 1 -- 172.21.15.84:0/2126530619 <== mon.2 v1:172.21.15.84:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4236868694 0 0) 0x7fa2f4005000 con 0x7fa2fc030660 2024-03-29T11:20:04.428 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:04.447+0000 7fa2ebfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:04.428 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:04.447+0000 7fa2ebfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:20:04.428 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:04.447+0000 7fa2ebfff640 10 monclient: dump: 2024-03-29T11:20:04.428 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T11:20:04.428 INFO:teuthology.misc.health.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:20:04.428 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T11:18:53.864025+0000 2024-03-29T11:20:04.428 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T11:18:53.864025+0000 2024-03-29T11:20:04.428 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:04.428 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:20:04.429 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T11:20:04.429 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T11:20:04.429 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T11:20:04.429 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:20:04.430 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:04.447+0000 7fa300ff9640 4 set_mon_vals no callback set 2024-03-29T11:20:04.430 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:04.450+0000 7fa303fff640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:20:04.430 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:04.450+0000 7fa303fff640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:20:04.430 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:04.450+0000 7fa303fff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-29T11:20:04.430 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:04.450+0000 7fa303fff640 1 -- 172.21.15.84:0/2126530619 --> v1:172.21.15.84:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa2e4005080 con 0x7fa2fc030660 2024-03-29T11:20:04.430 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:04.451+0000 7fa2ebfff640 1 -- 172.21.15.84:0/2126530619 <== mon.2 v1:172.21.15.84:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4033942909 0 0) 0x7fa2f4005160 con 0x7fa2fc030660 2024-03-29T11:20:04.430 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:04.452+0000 7fa2ebfff640 1 -- 172.21.15.84:0/2126530619 <== mon.2 v1:172.21.15.84:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2808139587 0 0) 0x7fa2f4084ca0 con 0x7fa2fc030660 2024-03-29T11:20:04.430 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:04.452+0000 7fa2ebfff640 10 client.4259.objecter ms_dispatch 0x7fa304000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:20:04.430 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:04.452+0000 7fa2ebfff640 3 client.4259.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:20:04.430 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:04.452+0000 7fa2ebfff640 3 client.4259.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:20:04.431 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:04.452+0000 7fa2ebfff640 20 client.4259.objecter dump_active .. 0 homeless 2024-03-29T11:20:04.431 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:04.453+0000 7fa3027fc640 10 monclient: get_auth_request con 0x7fa2f006dc90 auth_method 0 2024-03-29T11:20:04.431 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:04.453+0000 7fa2ebfff640 10 client.4259.objecter ms_handle_connect 0x7fa2f006dc90 2024-03-29T11:20:04.436 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:04.460+0000 7fa2ebfff640 1 -- 172.21.15.84:0/2126530619 <== mon.2 v1:172.21.15.84:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fa2f4050580 con 0x7fa2fc030660 2024-03-29T11:20:04.436 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:04.460+0000 7fa2ebfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:20:04.437 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:04.460+0000 7fa2ebfff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:20:04.756 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:04.779+0000 7fa303fff640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:20:04.756 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:04.779+0000 7fa303fff640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:20:04.756 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:04.779+0000 7fa303fff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-29T11:20:04.756 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:04.779+0000 7fa303fff640 1 -- 172.21.15.84:0/2126530619 --> v1:172.21.15.84:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa2e4005330 con 0x7fa2fc030660 2024-03-29T11:20:04.758 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:04.781+0000 7fa2ebfff640 1 -- 172.21.15.84:0/2126530619 <== mon.2 v1:172.21.15.84:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4209477549) 0x7fa2f40524b0 con 0x7fa2fc030660 2024-03-29T11:20:04.758 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:04.781+0000 7fa2ebfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:20:04.758 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:04.781+0000 7fa2ebfff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:20:04.759 INFO:teuthology.misc.health.smithi084.stdout: 2024-03-29T11:20:04.760 INFO:teuthology.misc.health.smithi084.stdout:{"epoch":10,"fsid":"2c3e002f-9eb0-4860-853b-4ef1ef562213","created":"2024-03-29T11:19:32.029169+0000","modified":"2024-03-29T11:19:34.200299+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:34.200299+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"5c39e276-2034-4716-bbc0-57a634eefa92","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"cd580e99-109f-4bf1-88c0-a5987bc8b9ed","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"11e60a5a-051d-453f-a190-66387cb35592","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"2c329fbe-bbc4-4030-bf87-d711cad04679","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f4213d4a-bd36-4433-91ae-d7e2b02481d3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"f0cf44ea-6055-48af-805b-aed858102ff9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"d895eaa0-cc0b-4ee4-9b5a-2dfcd89a60b7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"9bc5a7bc-8f52-4db9-96d6-de5e68d52eaa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:20:04.762 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:04.785+0000 7fa303fff640 10 librados: watch_flush enter 2024-03-29T11:20:04.762 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:04.785+0000 7fa303fff640 10 librados: watch_flush exit 2024-03-29T11:20:04.762 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:04.785+0000 7fa303fff640 20 client.4259.objecter shutdown clearing up homeless session... 2024-03-29T11:20:04.762 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:04.785+0000 7fa303fff640 10 client.4259.objecter successfully canceled tick 2024-03-29T11:20:04.762 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:04.785+0000 7fa303fff640 1 -- 172.21.15.84:0/2126530619 >> v1:172.21.15.179:6816/2981134980 conn(0x7fa2f006dc90 legacy=0x7fa2f0070150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:04.762 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:04.785+0000 7fa303fff640 10 monclient: shutdown 2024-03-29T11:20:04.762 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:04.785+0000 7fa303fff640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:20:04.762 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:04.785+0000 7fa303fff640 1 -- 172.21.15.84:0/2126530619 >> v1:172.21.15.84:6790/0 conn(0x7fa2fc030660 legacy=0x7fa2fc15c900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:04.762 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:04.786+0000 7fa303fff640 1 -- 172.21.15.84:0/2126530619 shutdown_connections 2024-03-29T11:20:04.763 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:04.786+0000 7fa303fff640 1 -- 172.21.15.84:0/2126530619 >> 172.21.15.84:0/2126530619 conn(0x7fa2fc005c00 msgr2=0x7fa2fc034090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:20:04.765 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:04.787+0000 7fa303fff640 1 -- 172.21.15.84:0/2126530619 shutdown_connections 2024-03-29T11:20:04.765 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:04.787+0000 7fa303fff640 1 -- 172.21.15.84:0/2126530619 wait complete. 2024-03-29T11:20:04.765 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:04.787+0000 7fa303fff640 1 librados: shutdown 2024-03-29T11:20:04.780 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:20:11.588 DEBUG:teuthology.orchestra.run.smithi084:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:20:11.745 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:11.767+0000 7febb03eb640 10 monclient: get_monmap_and_config 2024-03-29T11:20:11.746 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:11.767+0000 7febb03eb640 10 monclient: build_initial_monmap 2024-03-29T11:20:11.746 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:11.767+0000 7febb03eb640 10 monclient: monmap: 2024-03-29T11:20:11.746 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-29T11:20:11.746 INFO:teuthology.misc.health.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:20:11.746 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T11:20:11.768784+0000 2024-03-29T11:20:11.747 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T11:20:11.768784+0000 2024-03-29T11:20:11.747 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-29T11:20:11.747 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:20:11.747 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-29T11:20:11.747 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.noname-c 2024-03-29T11:20:11.747 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-b 2024-03-29T11:20:11.748 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:20:11.748 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:11.769+0000 7febb03eb640 1 Processor -- start 2024-03-29T11:20:11.751 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:11.770+0000 7febb03eb640 1 -- start start 2024-03-29T11:20:11.751 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:11.770+0000 7febb03eb640 10 monclient: init 2024-03-29T11:20:11.751 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:11.771+0000 7febb03eb640 10 monclient: _reopen_session rank -1 2024-03-29T11:20:11.751 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:11.771+0000 7febb03eb640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:20:11.751 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:11.771+0000 7febb03eb640 10 monclient(hunting): picked mon.noname-c con 0x7feba80a9ac0 addr v1:172.21.15.179:6789/0 2024-03-29T11:20:11.752 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:11.771+0000 7febb03eb640 10 monclient(hunting): picked mon.noname-a con 0x7feba813ee90 addr v1:172.21.15.84:6789/0 2024-03-29T11:20:11.752 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:11.771+0000 7febb03eb640 10 monclient(hunting): picked mon.noname-b con 0x7feba80af900 addr v1:172.21.15.84:6790/0 2024-03-29T11:20:11.752 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:11.771+0000 7febb03eb640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feba80afd10 con 0x7feba813ee90 2024-03-29T11:20:11.752 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:11.771+0000 7febb03eb640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feba80af6f0 con 0x7feba80a9ac0 2024-03-29T11:20:11.752 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:11.771+0000 7febb03eb640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feba80aac80 con 0x7feba80af900 2024-03-29T11:20:11.752 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:11.771+0000 7febb03eb640 10 monclient(hunting): _renew_subs 2024-03-29T11:20:11.752 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:11.771+0000 7febb03eb640 10 monclient(hunting): authenticate will time out at 873.065248s 2024-03-29T11:20:11.752 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:11.774+0000 7febae187640 1 --1- >> v1:172.21.15.179:6789/0 conn(0x7feba80a9ac0 0x7feba80adfe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.179:6789/0 says I am v1:172.21.15.84:47175/0 (socket says 172.21.15.84:47175) 2024-03-29T11:20:11.753 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:11.774+0000 7febae187640 1 -- 172.21.15.84:0/3711146277 learned_addr learned my addr 172.21.15.84:0/3711146277 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-29T11:20:11.753 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:11.776+0000 7febad185640 1 -- 172.21.15.84:0/3711146277 <== mon.1 v1:172.21.15.179:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1278822815 0 0) 0x7feba80af6f0 con 0x7feba80a9ac0 2024-03-29T11:20:11.753 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:11.776+0000 7febad185640 10 monclient(hunting): _negotiate 2024-03-29T11:20:11.753 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:11.776+0000 7febad185640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:11.753 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:11.776+0000 7febad185640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:11.753 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:11.776+0000 7febad185640 10 monclient(hunting): my global_id is 4231 2024-03-29T11:20:11.753 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:11.776+0000 7febad185640 1 -- 172.21.15.84:0/3711146277 --> v1:172.21.15.179:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feb94003520 con 0x7feba80a9ac0 2024-03-29T11:20:11.755 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:11.777+0000 7febad185640 1 -- 172.21.15.84:0/3711146277 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1841113865 0 0) 0x7feba80aac80 con 0x7feba80af900 2024-03-29T11:20:11.755 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:11.777+0000 7febad185640 10 monclient(hunting): _negotiate 2024-03-29T11:20:11.755 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:11.777+0000 7febad185640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:11.755 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:11.777+0000 7febad185640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:11.755 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:11.777+0000 7febad185640 10 monclient(hunting): my global_id is 4262 2024-03-29T11:20:11.755 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:11.777+0000 7febad185640 1 -- 172.21.15.84:0/3711146277 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feba80af6f0 con 0x7feba80af900 2024-03-29T11:20:11.755 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:11.777+0000 7febad185640 1 -- 172.21.15.84:0/3711146277 <== mon.1 v1:172.21.15.179:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2711227037 0 0) 0x7feb94003520 con 0x7feba80a9ac0 2024-03-29T11:20:11.755 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:11.777+0000 7febad185640 1 -- 172.21.15.84:0/3711146277 >> v1:172.21.15.84:6790/0 conn(0x7feba80af900 legacy=0x7feba8149b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:11.755 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:11.777+0000 7febad185640 1 -- 172.21.15.84:0/3711146277 >> v1:172.21.15.84:6789/0 conn(0x7feba813ee90 legacy=0x7feba813b5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:11.755 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:11.778+0000 7febad185640 10 monclient: _finish_hunting 0 2024-03-29T11:20:11.755 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:11.778+0000 7febad185640 1 monclient: found mon.noname-c 2024-03-29T11:20:11.756 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:11.778+0000 7febad185640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:20:11.756 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:11.778+0000 7febad185640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.179:6789/0 2024-03-29T11:20:11.756 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:11.778+0000 7febad185640 1 -- 172.21.15.84:0/3711146277 --> v1:172.21.15.179:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feba80aaf60 con 0x7feba80a9ac0 2024-03-29T11:20:11.756 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:11.778+0000 7febad185640 10 monclient: _finish_auth 0 2024-03-29T11:20:11.756 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:11.778+0000 7febad185640 10 monclient: _check_auth_tickets 2024-03-29T11:20:11.756 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:11.778+0000 7febad185640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:20:11.756 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:11.778+0000 7febb03eb640 5 monclient: authenticate success, global_id 4231 2024-03-29T11:20:11.756 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:11.778+0000 7febb03eb640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:20:11.756 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:11.778+0000 7febad185640 1 -- 172.21.15.84:0/3711146277 <== mon.1 v1:172.21.15.179:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4236868694 0 0) 0x7feba0001990 con 0x7feba80a9ac0 2024-03-29T11:20:11.756 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:11.778+0000 7febad185640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:11.756 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:11.778+0000 7febad185640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:20:11.756 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:11.778+0000 7febad185640 10 monclient: dump: 2024-03-29T11:20:11.756 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T11:20:11.756 INFO:teuthology.misc.health.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:20:11.756 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T11:18:53.864025+0000 2024-03-29T11:20:11.757 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T11:18:53.864025+0000 2024-03-29T11:20:11.757 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:11.757 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:20:11.757 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T11:20:11.757 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T11:20:11.757 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T11:20:11.757 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:20:11.757 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:11.778+0000 7febb03eb640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:20:11.757 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:11.780+0000 7febad185640 1 -- 172.21.15.84:0/3711146277 <== mon.1 v1:172.21.15.179:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7feba0001bb0 con 0x7feba80a9ac0 2024-03-29T11:20:11.758 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:11.780+0000 7febad185640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:20:11.760 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:11.780+0000 7febb03eb640 10 monclient: get_monmap_and_config success 2024-03-29T11:20:11.760 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:11.780+0000 7febb03eb640 4 set_mon_vals no callback set 2024-03-29T11:20:11.760 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:11.782+0000 7febad185640 1 -- 172.21.15.84:0/3711146277 <== mon.1 v1:172.21.15.179:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4236868694 0 0) 0x7feba0001f40 con 0x7feba80a9ac0 2024-03-29T11:20:11.760 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:11.782+0000 7febad185640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:11.760 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:11.782+0000 7febad185640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:20:11.760 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:11.782+0000 7febad185640 10 monclient: dump: 2024-03-29T11:20:11.760 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T11:20:11.760 INFO:teuthology.misc.health.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:20:11.761 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T11:18:53.864025+0000 2024-03-29T11:20:11.761 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T11:18:53.864025+0000 2024-03-29T11:20:11.761 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:11.761 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:20:11.761 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T11:20:11.761 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T11:20:11.761 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T11:20:11.761 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:20:11.761 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:11.782+0000 7febb03eb640 10 monclient: shutdown 2024-03-29T11:20:11.761 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:11.782+0000 7febb03eb640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:20:11.761 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:11.782+0000 7febb03eb640 1 -- 172.21.15.84:0/3711146277 >> v1:172.21.15.179:6789/0 conn(0x7feba80a9ac0 legacy=0x7feba80adfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:11.761 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:11.783+0000 7febb03eb640 1 -- 172.21.15.84:0/3711146277 shutdown_connections 2024-03-29T11:20:11.761 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:11.783+0000 7febb03eb640 1 -- 172.21.15.84:0/3711146277 >> 172.21.15.84:0/3711146277 conn(0x7feba80b35e0 msgr2=0x7feba80b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:20:11.761 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:11.783+0000 7febb03eb640 1 -- 172.21.15.84:0/3711146277 shutdown_connections 2024-03-29T11:20:11.761 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:11.783+0000 7febb03eb640 1 -- 172.21.15.84:0/3711146277 wait complete. 2024-03-29T11:20:11.762 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:11.784+0000 7febb03eb640 10 monclient: build_initial_monmap 2024-03-29T11:20:11.762 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:11.784+0000 7febb03eb640 10 monclient: monmap: 2024-03-29T11:20:11.762 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-29T11:20:11.762 INFO:teuthology.misc.health.smithi084.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:20:11.762 INFO:teuthology.misc.health.smithi084.stderr:last_changed 0.000000 2024-03-29T11:20:11.762 INFO:teuthology.misc.health.smithi084.stderr:created 0.000000 2024-03-29T11:20:11.762 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-29T11:20:11.762 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:20:11.763 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-29T11:20:11.763 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.noname-b 2024-03-29T11:20:11.763 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-c 2024-03-29T11:20:11.763 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:20:11.763 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:11.784+0000 7febb03eb640 1 librados: starting msgr at 2024-03-29T11:20:11.763 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:11.784+0000 7febb03eb640 1 librados: starting objecter 2024-03-29T11:20:11.763 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:11.784+0000 7febb03eb640 1 Processor -- start 2024-03-29T11:20:11.763 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:11.785+0000 7febb03eb640 1 -- start start 2024-03-29T11:20:11.763 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:11.785+0000 7febb03eb640 1 librados: setting wanted keys 2024-03-29T11:20:11.763 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:11.785+0000 7febb03eb640 1 librados: calling monclient init 2024-03-29T11:20:11.763 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:11.785+0000 7febb03eb640 10 monclient: init 2024-03-29T11:20:11.766 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:11.785+0000 7febb03eb640 10 monclient: _reopen_session rank -1 2024-03-29T11:20:11.766 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:11.785+0000 7febb03eb640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:20:11.766 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:11.785+0000 7febb03eb640 10 monclient(hunting): picked mon.noname-b con 0x7feba80a9ac0 addr v1:172.21.15.179:6789/0 2024-03-29T11:20:11.766 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:11.785+0000 7febb03eb640 10 monclient(hunting): picked mon.noname-a con 0x7feba80af900 addr v1:172.21.15.84:6789/0 2024-03-29T11:20:11.766 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:11.785+0000 7febb03eb640 10 monclient(hunting): picked mon.noname-c con 0x7feba813ee90 addr v1:172.21.15.84:6790/0 2024-03-29T11:20:11.766 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:11.785+0000 7febb03eb640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feba813fb70 con 0x7feba80af900 2024-03-29T11:20:11.766 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:11.785+0000 7febb03eb640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feba813a220 con 0x7feba80a9ac0 2024-03-29T11:20:11.766 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:11.785+0000 7febb03eb640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feba813f5c0 con 0x7feba813ee90 2024-03-29T11:20:11.766 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:11.785+0000 7febb03eb640 10 monclient(hunting): _renew_subs 2024-03-29T11:20:11.766 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:11.785+0000 7febb03eb640 10 monclient(hunting): authenticate will time out at 873.079971s 2024-03-29T11:20:11.767 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:11.790+0000 7febad986640 1 --1- >> v1:172.21.15.84:6789/0 conn(0x7feba80af900 0x7feba8142070 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.84:6789/0 says I am v1:172.21.15.84:47087/0 (socket says 172.21.15.84:47087) 2024-03-29T11:20:11.767 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:11.790+0000 7febad986640 1 -- 172.21.15.84:0/2171971835 learned_addr learned my addr 172.21.15.84:0/2171971835 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-29T11:20:11.768 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:11.790+0000 7feb9effd640 10 client.?.objecter ms_handle_connect 0x7feba80af900 2024-03-29T11:20:11.768 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:11.790+0000 7feb9effd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:20:11.768 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:11.790+0000 7feb9effd640 1 -- 172.21.15.84:0/2171971835 <== mon.0 v1:172.21.15.84:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2979207417 0 0) 0x7feba813fb70 con 0x7feba80af900 2024-03-29T11:20:11.768 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:11.790+0000 7feb9effd640 10 monclient(hunting): _negotiate 2024-03-29T11:20:11.768 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:11.790+0000 7feb9effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:11.768 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:11.790+0000 7feb9effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:11.768 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:11.790+0000 7feb9effd640 10 monclient(hunting): my global_id is 4260 2024-03-29T11:20:11.768 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:11.790+0000 7feb9effd640 1 -- 172.21.15.84:0/2171971835 --> v1:172.21.15.84:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feb8c004eb0 con 0x7feba80af900 2024-03-29T11:20:11.768 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:11.791+0000 7feb9effd640 1 -- 172.21.15.84:0/2171971835 <== mon.0 v1:172.21.15.84:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2412925856 0 0) 0x7feb8c004eb0 con 0x7feba80af900 2024-03-29T11:20:11.768 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:11.791+0000 7feb9effd640 1 -- 172.21.15.84:0/2171971835 --> v1:172.21.15.84:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7feba813fb70 con 0x7feba80af900 2024-03-29T11:20:11.768 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:11.791+0000 7feb9effd640 1 -- 172.21.15.84:0/2171971835 <== mon.0 v1:172.21.15.84:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4236868694 0 0) 0x7feb98001990 con 0x7feba80af900 2024-03-29T11:20:11.768 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:11.791+0000 7feb9effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:11.768 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:11.791+0000 7feb9effd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:20:11.769 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:11.791+0000 7feb9effd640 10 monclient(hunting): dump: 2024-03-29T11:20:11.769 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T11:20:11.769 INFO:teuthology.misc.health.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:20:11.769 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T11:18:53.864025+0000 2024-03-29T11:20:11.769 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T11:18:53.864025+0000 2024-03-29T11:20:11.769 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:11.769 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:20:11.769 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T11:20:11.769 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T11:20:11.769 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T11:20:11.769 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:20:11.769 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:11.791+0000 7feb9effd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:20:11.769 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:11.791+0000 7feb9effd640 1 -- 172.21.15.84:0/2171971835 <== mon.0 v1:172.21.15.84:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 780232045 0 0) 0x7feba813fb70 con 0x7feba80af900 2024-03-29T11:20:11.769 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:11.791+0000 7feb9effd640 1 -- 172.21.15.84:0/2171971835 >> v1:172.21.15.84:6790/0 conn(0x7feba813ee90 legacy=0x7feba8135820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:11.769 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:11.791+0000 7feb9effd640 1 -- 172.21.15.84:0/2171971835 >> v1:172.21.15.179:6789/0 conn(0x7feba80a9ac0 legacy=0x7feba813fe80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:20:11.771 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:11.793+0000 7feb9effd640 10 monclient: _finish_hunting 0 2024-03-29T11:20:11.771 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:11.793+0000 7feb9effd640 1 monclient: found mon.a 2024-03-29T11:20:11.771 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:11.793+0000 7feb9effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:20:11.771 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:11.793+0000 7feb9effd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.84:6789/0 2024-03-29T11:20:11.771 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:11.793+0000 7feb9effd640 1 -- 172.21.15.84:0/2171971835 --> v1:172.21.15.84:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feba813f750 con 0x7feba80af900 2024-03-29T11:20:11.771 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:11.793+0000 7feb9effd640 10 monclient: _finish_auth 0 2024-03-29T11:20:11.771 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:11.793+0000 7feb9effd640 10 monclient: _check_auth_tickets 2024-03-29T11:20:11.772 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:11.793+0000 7feb9effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:20:11.772 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:11.794+0000 7febb03eb640 5 monclient: authenticate success, global_id 4260 2024-03-29T11:20:11.772 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:11.794+0000 7febb03eb640 10 monclient: _renew_subs 2024-03-29T11:20:11.772 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:11.794+0000 7febb03eb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.84:6789/0 2024-03-29T11:20:11.772 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:11.794+0000 7febb03eb640 1 -- 172.21.15.84:0/2171971835 --> v1:172.21.15.84:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feba80723d0 con 0x7feba80af900 2024-03-29T11:20:11.772 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:11.794+0000 7febb03eb640 10 client.4260.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:20:11.772 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:11.794+0000 7febb03eb640 10 monclient: _renew_subs 2024-03-29T11:20:11.772 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:11.794+0000 7febb03eb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.84:6789/0 2024-03-29T11:20:11.772 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:11.794+0000 7febb03eb640 1 -- 172.21.15.84:0/2171971835 --> v1:172.21.15.84:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7feba8136110 con 0x7feba80af900 2024-03-29T11:20:11.772 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:11.794+0000 7febb03eb640 1 librados: init done 2024-03-29T11:20:11.773 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:11.795+0000 7feb9effd640 1 -- 172.21.15.84:0/2171971835 <== mon.0 v1:172.21.15.84:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7feb98001bf0 con 0x7feba80af900 2024-03-29T11:20:11.773 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:11.795+0000 7feb9effd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:20:11.773 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:11.795+0000 7feb9effd640 1 -- 172.21.15.84:0/2171971835 <== mon.0 v1:172.21.15.84:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4236868694 0 0) 0x7feb98003a60 con 0x7feba80af900 2024-03-29T11:20:11.773 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:11.795+0000 7feb9effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:11.773 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:11.795+0000 7feb9effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:20:11.774 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:11.795+0000 7feb9effd640 10 monclient: dump: 2024-03-29T11:20:11.774 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T11:20:11.774 INFO:teuthology.misc.health.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:20:11.774 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T11:18:53.864025+0000 2024-03-29T11:20:11.774 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T11:18:53.864025+0000 2024-03-29T11:20:11.774 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:11.774 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:20:11.774 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T11:20:11.774 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T11:20:11.774 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T11:20:11.774 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:20:11.774 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:11.795+0000 7febac984640 4 set_mon_vals no callback set 2024-03-29T11:20:11.774 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:11.796+0000 7febb03eb640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:20:11.775 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:11.797+0000 7febb03eb640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:20:11.775 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:11.797+0000 7febb03eb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.84:6789/0 2024-03-29T11:20:11.776 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:11.797+0000 7febb03eb640 1 -- 172.21.15.84:0/2171971835 --> v1:172.21.15.84:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feb90005080 con 0x7feba80af900 2024-03-29T11:20:11.776 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:11.798+0000 7feb9effd640 1 -- 172.21.15.84:0/2171971835 <== mon.0 v1:172.21.15.84:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4033942909 0 0) 0x7feb98003bc0 con 0x7feba80af900 2024-03-29T11:20:11.776 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:11.798+0000 7feb9effd640 1 -- 172.21.15.84:0/2171971835 <== mon.0 v1:172.21.15.84:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2808139587 0 0) 0x7feb98084c80 con 0x7feba80af900 2024-03-29T11:20:11.776 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:11.798+0000 7feb9effd640 10 client.4260.objecter ms_dispatch 0x7feba8000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:20:11.776 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:11.798+0000 7feb9effd640 3 client.4260.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:20:11.776 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:11.798+0000 7feb9effd640 3 client.4260.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:20:11.776 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:11.799+0000 7feb9effd640 20 client.4260.objecter dump_active .. 0 homeless 2024-03-29T11:20:11.777 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:11.800+0000 7febae187640 10 monclient: get_auth_request con 0x7feb8c06db10 auth_method 0 2024-03-29T11:20:11.778 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:11.801+0000 7feb9effd640 10 client.4260.objecter ms_handle_connect 0x7feb8c06db10 2024-03-29T11:20:11.787 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:11.811+0000 7feb9effd640 1 -- 172.21.15.84:0/2171971835 <== mon.0 v1:172.21.15.84:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7feb98050390 con 0x7feba80af900 2024-03-29T11:20:11.787 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:11.811+0000 7feb9effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:20:11.787 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:11.811+0000 7feb9effd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:20:12.129 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:12.152+0000 7febb03eb640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:20:12.129 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:12.152+0000 7febb03eb640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:20:12.129 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:12.152+0000 7febb03eb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.84:6789/0 2024-03-29T11:20:12.129 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:12.152+0000 7febb03eb640 1 -- 172.21.15.84:0/2171971835 --> v1:172.21.15.84:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7feb90005330 con 0x7feba80af900 2024-03-29T11:20:12.130 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:12.153+0000 7feb9effd640 1 -- 172.21.15.84:0/2171971835 <== mon.0 v1:172.21.15.84:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4209477549) 0x7feb98052490 con 0x7feba80af900 2024-03-29T11:20:12.130 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:12.153+0000 7feb9effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:20:12.130 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:12.153+0000 7feb9effd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:20:12.131 INFO:teuthology.misc.health.smithi084.stdout: 2024-03-29T11:20:12.131 INFO:teuthology.misc.health.smithi084.stdout:{"epoch":10,"fsid":"2c3e002f-9eb0-4860-853b-4ef1ef562213","created":"2024-03-29T11:19:32.029169+0000","modified":"2024-03-29T11:19:34.200299+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:34.200299+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"5c39e276-2034-4716-bbc0-57a634eefa92","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"cd580e99-109f-4bf1-88c0-a5987bc8b9ed","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"11e60a5a-051d-453f-a190-66387cb35592","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"2c329fbe-bbc4-4030-bf87-d711cad04679","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f4213d4a-bd36-4433-91ae-d7e2b02481d3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"f0cf44ea-6055-48af-805b-aed858102ff9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"d895eaa0-cc0b-4ee4-9b5a-2dfcd89a60b7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"9bc5a7bc-8f52-4db9-96d6-de5e68d52eaa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:20:12.133 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:12.157+0000 7febb03eb640 10 librados: watch_flush enter 2024-03-29T11:20:12.133 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:12.157+0000 7febb03eb640 10 librados: watch_flush exit 2024-03-29T11:20:12.133 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:12.157+0000 7febb03eb640 20 client.4260.objecter shutdown clearing up homeless session... 2024-03-29T11:20:12.134 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:12.157+0000 7febb03eb640 10 client.4260.objecter successfully canceled tick 2024-03-29T11:20:12.134 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:12.157+0000 7febb03eb640 1 -- 172.21.15.84:0/2171971835 >> v1:172.21.15.179:6816/2981134980 conn(0x7feb8c06db10 legacy=0x7feb8c06ffd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:12.134 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:12.157+0000 7febb03eb640 10 monclient: shutdown 2024-03-29T11:20:12.135 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:12.157+0000 7febb03eb640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:20:12.135 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:12.157+0000 7febb03eb640 1 -- 172.21.15.84:0/2171971835 >> v1:172.21.15.84:6789/0 conn(0x7feba80af900 legacy=0x7feba8142070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:12.135 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:12.157+0000 7febb03eb640 1 -- 172.21.15.84:0/2171971835 shutdown_connections 2024-03-29T11:20:12.135 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:12.157+0000 7febb03eb640 1 -- 172.21.15.84:0/2171971835 >> 172.21.15.84:0/2171971835 conn(0x7feba80b35e0 msgr2=0x7feba8136780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:20:12.135 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:12.158+0000 7febb03eb640 1 -- 172.21.15.84:0/2171971835 shutdown_connections 2024-03-29T11:20:12.135 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:12.158+0000 7febb03eb640 1 -- 172.21.15.84:0/2171971835 wait complete. 2024-03-29T11:20:12.135 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:12.158+0000 7febb03eb640 1 librados: shutdown 2024-03-29T11:20:12.152 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:20:18.958 DEBUG:teuthology.orchestra.run.smithi084:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:20:19.107 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:19.129+0000 7fd22c2a8640 10 monclient: get_monmap_and_config 2024-03-29T11:20:19.108 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:19.129+0000 7fd22c2a8640 10 monclient: build_initial_monmap 2024-03-29T11:20:19.108 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:19.129+0000 7fd22c2a8640 10 monclient: monmap: 2024-03-29T11:20:19.108 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-29T11:20:19.108 INFO:teuthology.misc.health.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:20:19.108 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T11:20:19.130147+0000 2024-03-29T11:20:19.108 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T11:20:19.130147+0000 2024-03-29T11:20:19.108 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-29T11:20:19.108 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:20:19.108 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-29T11:20:19.108 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.noname-c 2024-03-29T11:20:19.108 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-b 2024-03-29T11:20:19.108 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:20:19.109 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:19.131+0000 7fd22c2a8640 1 Processor -- start 2024-03-29T11:20:19.109 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:19.132+0000 7fd22c2a8640 1 -- start start 2024-03-29T11:20:19.109 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:19.132+0000 7fd22c2a8640 10 monclient: init 2024-03-29T11:20:19.110 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:19.133+0000 7fd22c2a8640 10 monclient: _reopen_session rank -1 2024-03-29T11:20:19.110 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:19.133+0000 7fd22c2a8640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:20:19.110 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:19.133+0000 7fd22c2a8640 10 monclient(hunting): picked mon.noname-a con 0x7fd2240ae250 addr v1:172.21.15.84:6789/0 2024-03-29T11:20:19.111 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:19.133+0000 7fd22c2a8640 10 monclient(hunting): picked mon.noname-b con 0x7fd22413f970 addr v1:172.21.15.84:6790/0 2024-03-29T11:20:19.111 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:19.133+0000 7fd22c2a8640 10 monclient(hunting): picked mon.noname-c con 0x7fd224142800 addr v1:172.21.15.179:6789/0 2024-03-29T11:20:19.111 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:19.133+0000 7fd22c2a8640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd2240afe00 con 0x7fd2240ae250 2024-03-29T11:20:19.111 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:19.133+0000 7fd22c2a8640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd224142c10 con 0x7fd224142800 2024-03-29T11:20:19.111 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:19.133+0000 7fd22c2a8640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd2240afb20 con 0x7fd22413f970 2024-03-29T11:20:19.111 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:19.133+0000 7fd22c2a8640 10 monclient(hunting): _renew_subs 2024-03-29T11:20:19.111 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:19.133+0000 7fd22c2a8640 10 monclient(hunting): authenticate will time out at 880.427552s 2024-03-29T11:20:19.113 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:19.137+0000 7fd22a845640 1 --1- >> v1:172.21.15.179:6789/0 conn(0x7fd224142800 0x7fd22414da10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.179:6789/0 says I am v1:172.21.15.84:37291/0 (socket says 172.21.15.84:37291) 2024-03-29T11:20:19.113 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:19.137+0000 7fd22a845640 1 -- 172.21.15.84:0/139473874 learned_addr learned my addr 172.21.15.84:0/139473874 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-29T11:20:19.116 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:19.138+0000 7fd229042640 1 -- 172.21.15.84:0/139473874 <== mon.1 v1:172.21.15.179:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3648700359 0 0) 0x7fd224142c10 con 0x7fd224142800 2024-03-29T11:20:19.116 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:19.138+0000 7fd229042640 10 monclient(hunting): _negotiate 2024-03-29T11:20:19.116 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:19.138+0000 7fd229042640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:19.116 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:19.138+0000 7fd229042640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:19.116 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:19.138+0000 7fd229042640 10 monclient(hunting): my global_id is 4234 2024-03-29T11:20:19.116 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:19.138+0000 7fd229042640 1 -- 172.21.15.84:0/139473874 --> v1:172.21.15.179:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd210003520 con 0x7fd224142800 2024-03-29T11:20:19.116 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:19.138+0000 7fd229042640 1 -- 172.21.15.84:0/139473874 <== mon.0 v1:172.21.15.84:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1597368026 0 0) 0x7fd2240afe00 con 0x7fd2240ae250 2024-03-29T11:20:19.116 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:19.139+0000 7fd229042640 10 monclient(hunting): _negotiate 2024-03-29T11:20:19.116 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:19.139+0000 7fd229042640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:19.117 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:19.139+0000 7fd229042640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:19.117 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:19.139+0000 7fd229042640 10 monclient(hunting): my global_id is 4263 2024-03-29T11:20:19.117 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:19.139+0000 7fd229042640 1 -- 172.21.15.84:0/139473874 --> v1:172.21.15.84:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd210003840 con 0x7fd2240ae250 2024-03-29T11:20:19.117 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:19.141+0000 7fd229042640 1 -- 172.21.15.84:0/139473874 <== mon.0 v1:172.21.15.84:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3710500035 0 0) 0x7fd210003840 con 0x7fd2240ae250 2024-03-29T11:20:19.117 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:19.141+0000 7fd229042640 1 -- 172.21.15.84:0/139473874 >> v1:172.21.15.84:6790/0 conn(0x7fd22413f970 legacy=0x7fd2241420f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:20:19.118 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:19.141+0000 7fd229042640 1 -- 172.21.15.84:0/139473874 >> v1:172.21.15.179:6789/0 conn(0x7fd224142800 legacy=0x7fd22414da10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:19.118 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:19.141+0000 7fd229042640 10 monclient: _finish_hunting 0 2024-03-29T11:20:19.118 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:19.141+0000 7fd229042640 1 monclient: found mon.noname-a 2024-03-29T11:20:19.118 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:19.141+0000 7fd229042640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:20:19.118 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:19.141+0000 7fd229042640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.84:6789/0 2024-03-29T11:20:19.118 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:19.141+0000 7fd229042640 1 -- 172.21.15.84:0/139473874 --> v1:172.21.15.84:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2240aab80 con 0x7fd2240ae250 2024-03-29T11:20:19.118 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:19.141+0000 7fd229042640 10 monclient: _finish_auth 0 2024-03-29T11:20:19.118 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:19.141+0000 7fd229042640 10 monclient: _check_auth_tickets 2024-03-29T11:20:19.118 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:19.141+0000 7fd229042640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:20:19.118 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:19.141+0000 7fd22c2a8640 5 monclient: authenticate success, global_id 4263 2024-03-29T11:20:19.118 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:19.141+0000 7fd22c2a8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:20:19.118 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:19.141+0000 7fd229042640 1 -- 172.21.15.84:0/139473874 <== mon.0 v1:172.21.15.84:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4236868694 0 0) 0x7fd214001560 con 0x7fd2240ae250 2024-03-29T11:20:19.118 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:19.141+0000 7fd229042640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:19.118 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:19.141+0000 7fd229042640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:20:19.119 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:19.141+0000 7fd229042640 10 monclient: dump: 2024-03-29T11:20:19.119 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T11:20:19.119 INFO:teuthology.misc.health.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:20:19.119 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T11:18:53.864025+0000 2024-03-29T11:20:19.119 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T11:18:53.864025+0000 2024-03-29T11:20:19.119 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:19.119 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:20:19.119 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T11:20:19.119 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T11:20:19.119 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T11:20:19.119 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:20:19.119 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:19.141+0000 7fd22c2a8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:20:19.120 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:19.142+0000 7fd229042640 1 -- 172.21.15.84:0/139473874 <== mon.0 v1:172.21.15.84:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd2140017b0 con 0x7fd2240ae250 2024-03-29T11:20:19.120 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:19.142+0000 7fd229042640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:20:19.120 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:19.142+0000 7fd229042640 1 -- 172.21.15.84:0/139473874 <== mon.0 v1:172.21.15.84:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4236868694 0 0) 0x7fd214001560 con 0x7fd2240ae250 2024-03-29T11:20:19.120 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:19.142+0000 7fd229042640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:19.120 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:19.142+0000 7fd229042640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:20:19.120 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:19.142+0000 7fd229042640 10 monclient: dump: 2024-03-29T11:20:19.121 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T11:20:19.121 INFO:teuthology.misc.health.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:20:19.121 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T11:18:53.864025+0000 2024-03-29T11:20:19.121 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T11:18:53.864025+0000 2024-03-29T11:20:19.121 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:19.121 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:20:19.121 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T11:20:19.121 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T11:20:19.121 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T11:20:19.121 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:20:19.121 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:19.142+0000 7fd22c2a8640 10 monclient: get_monmap_and_config success 2024-03-29T11:20:19.121 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:19.142+0000 7fd22c2a8640 4 set_mon_vals no callback set 2024-03-29T11:20:19.121 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:19.143+0000 7fd22c2a8640 10 monclient: shutdown 2024-03-29T11:20:19.121 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:19.143+0000 7fd22c2a8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:20:19.121 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:19.143+0000 7fd22c2a8640 1 -- 172.21.15.84:0/139473874 >> v1:172.21.15.84:6789/0 conn(0x7fd2240ae250 legacy=0x7fd22413f260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:19.123 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:19.146+0000 7fd22c2a8640 1 -- 172.21.15.84:0/139473874 shutdown_connections 2024-03-29T11:20:19.123 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:19.146+0000 7fd22c2a8640 1 -- 172.21.15.84:0/139473874 >> 172.21.15.84:0/139473874 conn(0x7fd2240b35e0 msgr2=0x7fd2240b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:20:19.126 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:19.150+0000 7fd22c2a8640 1 -- 172.21.15.84:0/139473874 shutdown_connections 2024-03-29T11:20:19.127 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:19.150+0000 7fd22c2a8640 1 -- 172.21.15.84:0/139473874 wait complete. 2024-03-29T11:20:19.127 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:19.151+0000 7fd22c2a8640 10 monclient: build_initial_monmap 2024-03-29T11:20:19.127 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:19.151+0000 7fd22c2a8640 10 monclient: monmap: 2024-03-29T11:20:19.127 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-29T11:20:19.127 INFO:teuthology.misc.health.smithi084.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:20:19.127 INFO:teuthology.misc.health.smithi084.stderr:last_changed 0.000000 2024-03-29T11:20:19.128 INFO:teuthology.misc.health.smithi084.stderr:created 0.000000 2024-03-29T11:20:19.128 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-29T11:20:19.128 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:20:19.128 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-29T11:20:19.128 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.noname-b 2024-03-29T11:20:19.128 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-c 2024-03-29T11:20:19.128 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:20:19.130 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:19.151+0000 7fd22c2a8640 1 librados: starting msgr at 2024-03-29T11:20:19.130 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:19.151+0000 7fd22c2a8640 1 librados: starting objecter 2024-03-29T11:20:19.130 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:19.151+0000 7fd22c2a8640 1 Processor -- start 2024-03-29T11:20:19.130 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:19.152+0000 7fd22c2a8640 1 -- start start 2024-03-29T11:20:19.130 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:19.152+0000 7fd22c2a8640 1 librados: setting wanted keys 2024-03-29T11:20:19.130 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:19.152+0000 7fd22c2a8640 1 librados: calling monclient init 2024-03-29T11:20:19.130 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:19.152+0000 7fd22c2a8640 10 monclient: init 2024-03-29T11:20:19.130 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:19.152+0000 7fd22c2a8640 10 monclient: _reopen_session rank -1 2024-03-29T11:20:19.130 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:19.152+0000 7fd22c2a8640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:20:19.130 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:19.152+0000 7fd22c2a8640 10 monclient(hunting): picked mon.noname-b con 0x7fd2240ae250 addr v1:172.21.15.179:6789/0 2024-03-29T11:20:19.131 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:19.152+0000 7fd22c2a8640 10 monclient(hunting): picked mon.noname-a con 0x7fd22413f970 addr v1:172.21.15.84:6789/0 2024-03-29T11:20:19.131 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:19.152+0000 7fd22c2a8640 10 monclient(hunting): picked mon.noname-c con 0x7fd224142800 addr v1:172.21.15.84:6790/0 2024-03-29T11:20:19.131 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:19.152+0000 7fd22c2a8640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd224144dc0 con 0x7fd22413f970 2024-03-29T11:20:19.131 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:19.152+0000 7fd22c2a8640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd224138d00 con 0x7fd2240ae250 2024-03-29T11:20:19.131 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:19.152+0000 7fd22c2a8640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd224145be0 con 0x7fd224142800 2024-03-29T11:20:19.131 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:19.152+0000 7fd22c2a8640 10 monclient(hunting): _renew_subs 2024-03-29T11:20:19.131 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:19.152+0000 7fd22c2a8640 10 monclient(hunting): authenticate will time out at 880.446979s 2024-03-29T11:20:19.142 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:19.159+0000 7fd22a044640 1 --1- >> v1:172.21.15.179:6789/0 conn(0x7fd2240ae250 0x7fd22413a940 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.179:6789/0 says I am v1:172.21.15.84:37427/0 (socket says 172.21.15.84:37427) 2024-03-29T11:20:19.142 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:19.159+0000 7fd22a845640 1 --1- >> v1:172.21.15.84:6790/0 conn(0x7fd224142800 0x7fd2241385f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.84:6790/0 says I am v1:172.21.15.84:42625/0 (socket says 172.21.15.84:42625) 2024-03-29T11:20:19.142 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:19.159+0000 7fd22a845640 1 -- 172.21.15.84:0/1604702526 learned_addr learned my addr 172.21.15.84:0/1604702526 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-29T11:20:19.142 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:19.159+0000 7fd21affd640 10 client.?.objecter ms_handle_connect 0x7fd224142800 2024-03-29T11:20:19.143 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:19.159+0000 7fd21affd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:20:19.143 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:19.160+0000 7fd21affd640 10 client.?.objecter ms_handle_connect 0x7fd2240ae250 2024-03-29T11:20:19.143 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:19.160+0000 7fd21affd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:20:19.143 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:19.160+0000 7fd21affd640 1 -- 172.21.15.84:0/1604702526 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2152419483 0 0) 0x7fd224145be0 con 0x7fd224142800 2024-03-29T11:20:19.143 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:19.160+0000 7fd21affd640 10 monclient(hunting): _negotiate 2024-03-29T11:20:19.143 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:19.160+0000 7fd21affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:19.143 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:19.160+0000 7fd21affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:19.143 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:19.160+0000 7fd21affd640 10 monclient(hunting): my global_id is 4265 2024-03-29T11:20:19.143 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:19.160+0000 7fd21affd640 1 -- 172.21.15.84:0/1604702526 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd208004eb0 con 0x7fd224142800 2024-03-29T11:20:19.143 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:19.160+0000 7fd21affd640 1 -- 172.21.15.84:0/1604702526 <== mon.1 v1:172.21.15.179:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 751833761 0 0) 0x7fd224138d00 con 0x7fd2240ae250 2024-03-29T11:20:19.143 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:19.160+0000 7fd21affd640 10 monclient(hunting): _negotiate 2024-03-29T11:20:19.143 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:19.160+0000 7fd21affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:19.144 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:19.160+0000 7fd21affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:19.144 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:19.160+0000 7fd21affd640 10 monclient(hunting): my global_id is 4237 2024-03-29T11:20:19.144 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:19.160+0000 7fd21affd640 1 -- 172.21.15.84:0/1604702526 --> v1:172.21.15.179:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd224145be0 con 0x7fd2240ae250 2024-03-29T11:20:19.144 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:19.161+0000 7fd21affd640 10 client.?.objecter ms_handle_connect 0x7fd22413f970 2024-03-29T11:20:19.144 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:19.161+0000 7fd21affd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:20:19.144 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:19.161+0000 7fd21affd640 1 -- 172.21.15.84:0/1604702526 <== mon.2 v1:172.21.15.84:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4129911741 0 0) 0x7fd208004eb0 con 0x7fd224142800 2024-03-29T11:20:19.144 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:19.161+0000 7fd21affd640 1 -- 172.21.15.84:0/1604702526 --> v1:172.21.15.84:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd224138d00 con 0x7fd224142800 2024-03-29T11:20:19.144 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:19.161+0000 7fd21affd640 1 -- 172.21.15.84:0/1604702526 <== mon.2 v1:172.21.15.84:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4236868694 0 0) 0x7fd21c001bf0 con 0x7fd224142800 2024-03-29T11:20:19.144 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:19.161+0000 7fd21affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:19.144 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:19.161+0000 7fd21affd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:20:19.144 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:19.161+0000 7fd21affd640 10 monclient(hunting): dump: 2024-03-29T11:20:19.144 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T11:20:19.144 INFO:teuthology.misc.health.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:20:19.145 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T11:18:53.864025+0000 2024-03-29T11:20:19.145 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T11:18:53.864025+0000 2024-03-29T11:20:19.145 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:19.145 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:20:19.145 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T11:20:19.145 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T11:20:19.145 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T11:20:19.145 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:20:19.145 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:19.161+0000 7fd21affd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:20:19.145 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:19.161+0000 7fd21affd640 1 -- 172.21.15.84:0/1604702526 <== mon.0 v1:172.21.15.84:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 707540708 0 0) 0x7fd224144dc0 con 0x7fd22413f970 2024-03-29T11:20:19.145 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:19.161+0000 7fd21affd640 10 monclient(hunting): _negotiate 2024-03-29T11:20:19.145 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:19.161+0000 7fd21affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:19.145 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:19.161+0000 7fd21affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:19.146 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:19.161+0000 7fd21affd640 10 monclient(hunting): my global_id is 4266 2024-03-29T11:20:19.146 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:19.162+0000 7fd21affd640 1 -- 172.21.15.84:0/1604702526 --> v1:172.21.15.84:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd208004eb0 con 0x7fd22413f970 2024-03-29T11:20:19.146 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:19.162+0000 7fd21affd640 1 -- 172.21.15.84:0/1604702526 <== mon.2 v1:172.21.15.84:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2061644433 0 0) 0x7fd224138d00 con 0x7fd224142800 2024-03-29T11:20:19.146 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:19.162+0000 7fd21affd640 1 -- 172.21.15.84:0/1604702526 >> v1:172.21.15.179:6789/0 conn(0x7fd2240ae250 legacy=0x7fd22413a940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:19.146 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:19.162+0000 7fd21affd640 1 -- 172.21.15.84:0/1604702526 >> v1:172.21.15.84:6789/0 conn(0x7fd22413f970 legacy=0x7fd224137ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:19.146 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:19.162+0000 7fd21affd640 10 monclient: _finish_hunting 0 2024-03-29T11:20:19.146 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:19.162+0000 7fd21affd640 1 monclient: found mon.c 2024-03-29T11:20:19.146 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:19.162+0000 7fd21affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:20:19.146 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:19.162+0000 7fd21affd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-29T11:20:19.146 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:19.162+0000 7fd21affd640 1 -- 172.21.15.84:0/1604702526 --> v1:172.21.15.84:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd224145ec0 con 0x7fd224142800 2024-03-29T11:20:19.146 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:19.162+0000 7fd21affd640 10 monclient: _finish_auth 0 2024-03-29T11:20:19.146 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:19.162+0000 7fd21affd640 10 monclient: _check_auth_tickets 2024-03-29T11:20:19.146 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:19.162+0000 7fd21affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:20:19.146 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:19.162+0000 7fd22c2a8640 5 monclient: authenticate success, global_id 4265 2024-03-29T11:20:19.147 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:19.162+0000 7fd22c2a8640 10 monclient: _renew_subs 2024-03-29T11:20:19.147 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:19.162+0000 7fd22c2a8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-29T11:20:19.147 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:19.162+0000 7fd22c2a8640 1 -- 172.21.15.84:0/1604702526 --> v1:172.21.15.84:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd2241461a0 con 0x7fd224142800 2024-03-29T11:20:19.147 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:19.162+0000 7fd22c2a8640 10 client.4265.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:20:19.147 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:19.162+0000 7fd22c2a8640 10 monclient: _renew_subs 2024-03-29T11:20:19.147 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:19.162+0000 7fd22c2a8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-29T11:20:19.147 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:19.162+0000 7fd22c2a8640 1 -- 172.21.15.84:0/1604702526 --> v1:172.21.15.84:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fd224145490 con 0x7fd224142800 2024-03-29T11:20:19.147 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:19.162+0000 7fd22c2a8640 1 librados: init done 2024-03-29T11:20:19.147 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:19.164+0000 7fd21affd640 1 -- 172.21.15.84:0/1604702526 <== mon.2 v1:172.21.15.84:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd21c0020e0 con 0x7fd224142800 2024-03-29T11:20:19.147 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:19.164+0000 7fd21affd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:20:19.147 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:19.164+0000 7fd21affd640 1 -- 172.21.15.84:0/1604702526 <== mon.2 v1:172.21.15.84:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4236868694 0 0) 0x7fd21c004d40 con 0x7fd224142800 2024-03-29T11:20:19.147 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:19.164+0000 7fd21affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:19.147 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:19.164+0000 7fd21affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:20:19.147 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:19.164+0000 7fd21affd640 10 monclient: dump: 2024-03-29T11:20:19.148 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T11:20:19.148 INFO:teuthology.misc.health.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:20:19.148 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T11:18:53.864025+0000 2024-03-29T11:20:19.148 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T11:18:53.864025+0000 2024-03-29T11:20:19.148 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:19.148 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:20:19.148 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T11:20:19.148 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T11:20:19.148 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T11:20:19.148 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:20:19.148 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:19.164+0000 7fd228841640 4 set_mon_vals no callback set 2024-03-29T11:20:19.148 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:19.164+0000 7fd21affd640 1 -- 172.21.15.84:0/1604702526 <== mon.2 v1:172.21.15.84:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4033942909 0 0) 0x7fd21c01add0 con 0x7fd224142800 2024-03-29T11:20:19.148 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:19.165+0000 7fd21affd640 1 -- 172.21.15.84:0/1604702526 <== mon.2 v1:172.21.15.84:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2808139587 0 0) 0x7fd21c084d10 con 0x7fd224142800 2024-03-29T11:20:19.149 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:19.165+0000 7fd21affd640 10 client.4265.objecter ms_dispatch 0x7fd224000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:20:19.149 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:19.165+0000 7fd21affd640 3 client.4265.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:20:19.149 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:19.165+0000 7fd21affd640 3 client.4265.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:20:19.149 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:19.165+0000 7fd21affd640 20 client.4265.objecter dump_active .. 0 homeless 2024-03-29T11:20:19.149 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:19.165+0000 7fd22c2a8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:20:19.150 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:19.166+0000 7fd22c2a8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:20:19.150 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:19.167+0000 7fd22c2a8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-29T11:20:19.150 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:19.167+0000 7fd22c2a8640 1 -- 172.21.15.84:0/1604702526 --> v1:172.21.15.84:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd20c005080 con 0x7fd224142800 2024-03-29T11:20:19.150 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:19.167+0000 7fd22a044640 10 monclient: get_auth_request con 0x7fd20806de20 auth_method 0 2024-03-29T11:20:19.150 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:19.170+0000 7fd21affd640 10 client.4265.objecter ms_handle_connect 0x7fd20806de20 2024-03-29T11:20:19.153 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:19.177+0000 7fd21affd640 1 -- 172.21.15.84:0/1604702526 <== mon.2 v1:172.21.15.84:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fd21c0505a0 con 0x7fd224142800 2024-03-29T11:20:19.153 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:19.177+0000 7fd21affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:20:19.153 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:19.177+0000 7fd21affd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:20:19.486 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:19.510+0000 7fd22c2a8640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:20:19.487 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:19.510+0000 7fd22c2a8640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:20:19.487 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:19.510+0000 7fd22c2a8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-29T11:20:19.487 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:19.510+0000 7fd22c2a8640 1 -- 172.21.15.84:0/1604702526 --> v1:172.21.15.84:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd20c005330 con 0x7fd224142800 2024-03-29T11:20:19.487 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:19.511+0000 7fd21affd640 1 -- 172.21.15.84:0/1604702526 <== mon.2 v1:172.21.15.84:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4209477549) 0x7fd21c0526a0 con 0x7fd224142800 2024-03-29T11:20:19.488 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:19.511+0000 7fd21affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:20:19.488 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:19.511+0000 7fd21affd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:20:19.488 INFO:teuthology.misc.health.smithi084.stdout: 2024-03-29T11:20:19.488 INFO:teuthology.misc.health.smithi084.stdout:{"epoch":10,"fsid":"2c3e002f-9eb0-4860-853b-4ef1ef562213","created":"2024-03-29T11:19:32.029169+0000","modified":"2024-03-29T11:19:34.200299+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:34.200299+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"5c39e276-2034-4716-bbc0-57a634eefa92","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"cd580e99-109f-4bf1-88c0-a5987bc8b9ed","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"11e60a5a-051d-453f-a190-66387cb35592","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"2c329fbe-bbc4-4030-bf87-d711cad04679","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f4213d4a-bd36-4433-91ae-d7e2b02481d3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"f0cf44ea-6055-48af-805b-aed858102ff9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"d895eaa0-cc0b-4ee4-9b5a-2dfcd89a60b7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"9bc5a7bc-8f52-4db9-96d6-de5e68d52eaa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:20:19.491 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:19.515+0000 7fd22c2a8640 10 librados: watch_flush enter 2024-03-29T11:20:19.491 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:19.515+0000 7fd22c2a8640 10 librados: watch_flush exit 2024-03-29T11:20:19.492 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:19.515+0000 7fd22c2a8640 20 client.4265.objecter shutdown clearing up homeless session... 2024-03-29T11:20:19.492 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:19.515+0000 7fd22c2a8640 10 client.4265.objecter successfully canceled tick 2024-03-29T11:20:19.492 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:19.515+0000 7fd22c2a8640 1 -- 172.21.15.84:0/1604702526 >> v1:172.21.15.179:6816/2981134980 conn(0x7fd20806de20 legacy=0x7fd2080702e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:19.492 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:19.515+0000 7fd22c2a8640 10 monclient: shutdown 2024-03-29T11:20:19.492 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:19.515+0000 7fd22c2a8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:20:19.492 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:19.515+0000 7fd22c2a8640 1 -- 172.21.15.84:0/1604702526 >> v1:172.21.15.84:6790/0 conn(0x7fd224142800 legacy=0x7fd2241385f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:19.492 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:19.515+0000 7fd22c2a8640 1 -- 172.21.15.84:0/1604702526 shutdown_connections 2024-03-29T11:20:19.492 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:19.515+0000 7fd22c2a8640 1 -- 172.21.15.84:0/1604702526 >> 172.21.15.84:0/1604702526 conn(0x7fd2240b35e0 msgr2=0x7fd2240a9900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:20:19.492 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:19.516+0000 7fd22c2a8640 1 -- 172.21.15.84:0/1604702526 shutdown_connections 2024-03-29T11:20:19.493 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:19.516+0000 7fd22c2a8640 1 -- 172.21.15.84:0/1604702526 wait complete. 2024-03-29T11:20:19.495 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:19.516+0000 7fd22c2a8640 1 librados: shutdown 2024-03-29T11:20:19.510 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:20:26.318 DEBUG:teuthology.orchestra.run.smithi084:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:20:26.475 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:26.498+0000 7f484bfa8640 10 monclient: get_monmap_and_config 2024-03-29T11:20:26.475 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:26.498+0000 7f484bfa8640 10 monclient: build_initial_monmap 2024-03-29T11:20:26.475 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:26.498+0000 7f484bfa8640 10 monclient: monmap: 2024-03-29T11:20:26.475 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-29T11:20:26.475 INFO:teuthology.misc.health.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:20:26.475 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T11:20:26.499386+0000 2024-03-29T11:20:26.475 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T11:20:26.499386+0000 2024-03-29T11:20:26.475 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-29T11:20:26.475 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:20:26.476 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-29T11:20:26.476 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.noname-c 2024-03-29T11:20:26.476 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-b 2024-03-29T11:20:26.476 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:20:26.476 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:26.499+0000 7f484bfa8640 1 Processor -- start 2024-03-29T11:20:26.482 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:26.500+0000 7f484bfa8640 1 -- start start 2024-03-29T11:20:26.482 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:26.500+0000 7f484bfa8640 10 monclient: init 2024-03-29T11:20:26.482 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:26.500+0000 7f484bfa8640 10 monclient: _reopen_session rank -1 2024-03-29T11:20:26.483 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:26.500+0000 7f484bfa8640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:20:26.483 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:26.500+0000 7f484bfa8640 10 monclient(hunting): picked mon.noname-c con 0x7f48440b0770 addr v1:172.21.15.179:6789/0 2024-03-29T11:20:26.483 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:26.500+0000 7f484bfa8640 10 monclient(hunting): picked mon.noname-a con 0x7f484414a700 addr v1:172.21.15.84:6789/0 2024-03-29T11:20:26.483 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:26.500+0000 7f484bfa8640 10 monclient(hunting): picked mon.noname-b con 0x7f48440ac5a0 addr v1:172.21.15.84:6790/0 2024-03-29T11:20:26.483 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:26.500+0000 7f484bfa8640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f48440aba70 con 0x7f484414a700 2024-03-29T11:20:26.483 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:26.501+0000 7f484bfa8640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f48440aecb0 con 0x7f48440b0770 2024-03-29T11:20:26.483 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:26.501+0000 7f484bfa8640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f48440af150 con 0x7f48440ac5a0 2024-03-29T11:20:26.485 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:26.501+0000 7f484bfa8640 10 monclient(hunting): _renew_subs 2024-03-29T11:20:26.485 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:26.501+0000 7f484bfa8640 10 monclient(hunting): authenticate will time out at 887.795127s 2024-03-29T11:20:26.485 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:26.503+0000 7f4849d44640 1 --1- >> v1:172.21.15.179:6789/0 conn(0x7f48440b0770 0x7f4844149ff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.179:6789/0 says I am v1:172.21.15.84:48367/0 (socket says 172.21.15.84:48367) 2024-03-29T11:20:26.485 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:26.503+0000 7f4849d44640 1 -- 172.21.15.84:0/3063620402 learned_addr learned my addr 172.21.15.84:0/3063620402 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-29T11:20:26.486 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:26.504+0000 7f4848d42640 1 -- 172.21.15.84:0/3063620402 <== mon.1 v1:172.21.15.179:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 395448446 0 0) 0x7f48440aecb0 con 0x7f48440b0770 2024-03-29T11:20:26.486 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:26.504+0000 7f4848d42640 10 monclient(hunting): _negotiate 2024-03-29T11:20:26.486 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:26.504+0000 7f4848d42640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:26.486 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:26.504+0000 7f4848d42640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:26.486 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:26.504+0000 7f4848d42640 10 monclient(hunting): my global_id is 4240 2024-03-29T11:20:26.486 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:26.504+0000 7f4848d42640 1 -- 172.21.15.84:0/3063620402 --> v1:172.21.15.179:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4830014530 con 0x7f48440b0770 2024-03-29T11:20:26.486 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:26.504+0000 7f4848d42640 1 -- 172.21.15.84:0/3063620402 <== mon.1 v1:172.21.15.179:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 608390537 0 0) 0x7f4830014530 con 0x7f48440b0770 2024-03-29T11:20:26.486 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:26.504+0000 7f4848d42640 1 -- 172.21.15.84:0/3063620402 >> v1:172.21.15.84:6790/0 conn(0x7f48440ac5a0 legacy=0x7f4844151f00 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:20:26.489 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:26.508+0000 7f4848d42640 1 -- 172.21.15.84:0/3063620402 >> v1:172.21.15.84:6789/0 conn(0x7f484414a700 legacy=0x7f48440abe90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:26.489 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:26.508+0000 7f4848d42640 10 monclient: _finish_hunting 0 2024-03-29T11:20:26.489 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:26.508+0000 7f4848d42640 1 monclient: found mon.noname-c 2024-03-29T11:20:26.489 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:26.508+0000 7f4848d42640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:20:26.490 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:26.508+0000 7f4848d42640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.179:6789/0 2024-03-29T11:20:26.490 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:26.508+0000 7f4848d42640 1 -- 172.21.15.84:0/3063620402 --> v1:172.21.15.179:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48440ac9d0 con 0x7f48440b0770 2024-03-29T11:20:26.490 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:26.508+0000 7f4848d42640 10 monclient: _finish_auth 0 2024-03-29T11:20:26.490 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:26.508+0000 7f4848d42640 10 monclient: _check_auth_tickets 2024-03-29T11:20:26.490 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:26.508+0000 7f4848d42640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:20:26.490 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:26.508+0000 7f4848d42640 1 -- 172.21.15.84:0/3063620402 <== mon.1 v1:172.21.15.179:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4236868694 0 0) 0x7f4834001670 con 0x7f48440b0770 2024-03-29T11:20:26.490 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:26.508+0000 7f4848d42640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:26.490 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:26.508+0000 7f4848d42640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:20:26.490 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:26.508+0000 7f4848d42640 10 monclient: dump: 2024-03-29T11:20:26.490 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T11:20:26.490 INFO:teuthology.misc.health.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:20:26.490 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T11:18:53.864025+0000 2024-03-29T11:20:26.490 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T11:18:53.864025+0000 2024-03-29T11:20:26.491 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:26.491 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:20:26.491 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T11:20:26.491 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T11:20:26.491 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T11:20:26.491 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:20:26.491 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:26.508+0000 7f4848d42640 1 -- 172.21.15.84:0/3063620402 <== mon.1 v1:172.21.15.179:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f48340018d0 con 0x7f48440b0770 2024-03-29T11:20:26.491 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:26.508+0000 7f4848d42640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:20:26.491 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:26.508+0000 7f4848d42640 1 -- 172.21.15.84:0/3063620402 <== mon.1 v1:172.21.15.179:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4236868694 0 0) 0x7f4834001d20 con 0x7f48440b0770 2024-03-29T11:20:26.491 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:26.508+0000 7f4848d42640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:26.491 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:26.508+0000 7f4848d42640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:20:26.491 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:26.508+0000 7f4848d42640 10 monclient: dump: 2024-03-29T11:20:26.491 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T11:20:26.491 INFO:teuthology.misc.health.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:20:26.491 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T11:18:53.864025+0000 2024-03-29T11:20:26.492 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T11:18:53.864025+0000 2024-03-29T11:20:26.492 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:26.492 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:20:26.492 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T11:20:26.492 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T11:20:26.492 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T11:20:26.492 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:20:26.492 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:26.508+0000 7f484bfa8640 5 monclient: authenticate success, global_id 4240 2024-03-29T11:20:26.492 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:26.508+0000 7f484bfa8640 10 monclient: get_monmap_and_config success 2024-03-29T11:20:26.492 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:26.508+0000 7f484bfa8640 4 set_mon_vals no callback set 2024-03-29T11:20:26.492 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:26.509+0000 7f484bfa8640 10 monclient: shutdown 2024-03-29T11:20:26.492 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:26.509+0000 7f484bfa8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:20:26.492 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:26.509+0000 7f484bfa8640 1 -- 172.21.15.84:0/3063620402 >> v1:172.21.15.179:6789/0 conn(0x7f48440b0770 legacy=0x7f4844149ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:26.492 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:26.511+0000 7f484bfa8640 1 -- 172.21.15.84:0/3063620402 shutdown_connections 2024-03-29T11:20:26.492 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:26.511+0000 7f484bfa8640 1 -- 172.21.15.84:0/3063620402 >> 172.21.15.84:0/3063620402 conn(0x7f484406b660 msgr2=0x7f484406ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:20:26.493 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:26.512+0000 7f484bfa8640 1 -- 172.21.15.84:0/3063620402 shutdown_connections 2024-03-29T11:20:26.493 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:26.512+0000 7f484bfa8640 1 -- 172.21.15.84:0/3063620402 wait complete. 2024-03-29T11:20:26.493 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:26.513+0000 7f484bfa8640 10 monclient: build_initial_monmap 2024-03-29T11:20:26.493 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:26.513+0000 7f484bfa8640 10 monclient: monmap: 2024-03-29T11:20:26.493 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-29T11:20:26.493 INFO:teuthology.misc.health.smithi084.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:20:26.493 INFO:teuthology.misc.health.smithi084.stderr:last_changed 0.000000 2024-03-29T11:20:26.493 INFO:teuthology.misc.health.smithi084.stderr:created 0.000000 2024-03-29T11:20:26.493 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-29T11:20:26.493 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:20:26.493 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-29T11:20:26.493 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.noname-b 2024-03-29T11:20:26.493 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-c 2024-03-29T11:20:26.493 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:20:26.494 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:26.513+0000 7f484bfa8640 1 librados: starting msgr at 2024-03-29T11:20:26.494 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:26.513+0000 7f484bfa8640 1 librados: starting objecter 2024-03-29T11:20:26.494 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:26.514+0000 7f484bfa8640 1 Processor -- start 2024-03-29T11:20:26.494 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:26.517+0000 7f484bfa8640 1 -- start start 2024-03-29T11:20:26.494 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:26.517+0000 7f484bfa8640 1 librados: setting wanted keys 2024-03-29T11:20:26.494 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:26.517+0000 7f484bfa8640 1 librados: calling monclient init 2024-03-29T11:20:26.495 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:26.517+0000 7f484bfa8640 10 monclient: init 2024-03-29T11:20:26.495 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:26.517+0000 7f484bfa8640 10 monclient: _reopen_session rank -1 2024-03-29T11:20:26.495 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:26.517+0000 7f484bfa8640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:20:26.498 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:26.521+0000 7f484bfa8640 10 monclient(hunting): picked mon.noname-b con 0x7f48440ac5a0 addr v1:172.21.15.179:6789/0 2024-03-29T11:20:26.498 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:26.521+0000 7f484bfa8640 10 monclient(hunting): picked mon.noname-c con 0x7f48440b0770 addr v1:172.21.15.84:6790/0 2024-03-29T11:20:26.498 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:26.521+0000 7f484bfa8640 10 monclient(hunting): picked mon.noname-a con 0x7f484414a700 addr v1:172.21.15.84:6789/0 2024-03-29T11:20:26.498 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:26.521+0000 7f484bfa8640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4844146d10 con 0x7f484414a700 2024-03-29T11:20:26.498 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:26.521+0000 7f484bfa8640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f48441460e0 con 0x7f48440ac5a0 2024-03-29T11:20:26.498 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:26.521+0000 7f484bfa8640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4844070860 con 0x7f48440b0770 2024-03-29T11:20:26.498 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:26.521+0000 7f484bfa8640 10 monclient(hunting): _renew_subs 2024-03-29T11:20:26.498 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:26.521+0000 7f484bfa8640 10 monclient(hunting): authenticate will time out at 887.815607s 2024-03-29T11:20:26.503 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:26.524+0000 7f484a545640 1 --1- >> v1:172.21.15.84:6789/0 conn(0x7f484414a700 0x7f4844146600 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.84:6789/0 says I am v1:172.21.15.84:58183/0 (socket says 172.21.15.84:58183) 2024-03-29T11:20:26.503 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:26.524+0000 7f484a545640 1 -- 172.21.15.84:0/4014126582 learned_addr learned my addr 172.21.15.84:0/4014126582 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-29T11:20:26.503 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:26.524+0000 7f483a7fc640 10 client.?.objecter ms_handle_connect 0x7f484414a700 2024-03-29T11:20:26.503 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:26.524+0000 7f483a7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:20:26.503 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:26.525+0000 7f483a7fc640 10 client.?.objecter ms_handle_connect 0x7f48440ac5a0 2024-03-29T11:20:26.503 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:26.525+0000 7f483a7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:20:26.503 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:26.525+0000 7f483a7fc640 10 client.?.objecter ms_handle_connect 0x7f48440b0770 2024-03-29T11:20:26.503 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:26.525+0000 7f483a7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:20:26.503 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:26.525+0000 7f483a7fc640 1 -- 172.21.15.84:0/4014126582 <== mon.1 v1:172.21.15.179:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3125482501 0 0) 0x7f48441460e0 con 0x7f48440ac5a0 2024-03-29T11:20:26.503 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:26.525+0000 7f483a7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:20:26.503 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:26.525+0000 7f483a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:26.503 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:26.525+0000 7f483a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:26.503 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:26.525+0000 7f483a7fc640 10 monclient(hunting): my global_id is 4243 2024-03-29T11:20:26.504 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:26.525+0000 7f483a7fc640 1 -- 172.21.15.84:0/4014126582 --> v1:172.21.15.179:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4828004eb0 con 0x7f48440ac5a0 2024-03-29T11:20:26.504 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:26.525+0000 7f483a7fc640 1 -- 172.21.15.84:0/4014126582 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3109828814 0 0) 0x7f4844070860 con 0x7f48440b0770 2024-03-29T11:20:26.504 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:26.525+0000 7f483a7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:20:26.504 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:26.525+0000 7f483a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:26.504 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:26.525+0000 7f483a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:26.504 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:26.525+0000 7f483a7fc640 10 monclient(hunting): my global_id is 4268 2024-03-29T11:20:26.504 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:26.525+0000 7f483a7fc640 1 -- 172.21.15.84:0/4014126582 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f48441460e0 con 0x7f48440b0770 2024-03-29T11:20:26.504 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:26.525+0000 7f483a7fc640 1 -- 172.21.15.84:0/4014126582 <== mon.2 v1:172.21.15.84:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 934799874 0 0) 0x7f48441460e0 con 0x7f48440b0770 2024-03-29T11:20:26.504 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:26.525+0000 7f483a7fc640 1 -- 172.21.15.84:0/4014126582 --> v1:172.21.15.84:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4828005490 con 0x7f48440b0770 2024-03-29T11:20:26.504 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:26.525+0000 7f483a7fc640 1 -- 172.21.15.84:0/4014126582 <== mon.2 v1:172.21.15.84:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4236868694 0 0) 0x7f4840001b80 con 0x7f48440b0770 2024-03-29T11:20:26.504 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:26.525+0000 7f483a7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:26.504 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:26.525+0000 7f483a7fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:20:26.504 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:26.525+0000 7f483a7fc640 10 monclient(hunting): dump: 2024-03-29T11:20:26.504 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T11:20:26.504 INFO:teuthology.misc.health.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:20:26.505 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T11:18:53.864025+0000 2024-03-29T11:20:26.505 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T11:18:53.864025+0000 2024-03-29T11:20:26.505 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:26.505 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:20:26.505 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T11:20:26.505 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T11:20:26.505 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T11:20:26.505 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:20:26.505 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:26.525+0000 7f483a7fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:20:26.507 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:26.526+0000 7f483a7fc640 1 -- 172.21.15.84:0/4014126582 <== mon.2 v1:172.21.15.84:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 357221446 0 0) 0x7f4828005490 con 0x7f48440b0770 2024-03-29T11:20:26.507 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:26.526+0000 7f483a7fc640 1 -- 172.21.15.84:0/4014126582 >> v1:172.21.15.179:6789/0 conn(0x7f48440ac5a0 legacy=0x7f48441457b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:26.507 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:26.526+0000 7f483a7fc640 1 -- 172.21.15.84:0/4014126582 >> v1:172.21.15.84:6789/0 conn(0x7f484414a700 legacy=0x7f4844146600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:26.507 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:26.526+0000 7f483a7fc640 10 monclient: _finish_hunting 0 2024-03-29T11:20:26.507 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:26.526+0000 7f483a7fc640 1 monclient: found mon.c 2024-03-29T11:20:26.507 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:26.526+0000 7f483a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:20:26.507 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:26.526+0000 7f483a7fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-29T11:20:26.508 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:26.526+0000 7f483a7fc640 1 -- 172.21.15.84:0/4014126582 --> v1:172.21.15.84:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48441d61f0 con 0x7f48440b0770 2024-03-29T11:20:26.508 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:26.526+0000 7f483a7fc640 10 monclient: _finish_auth 0 2024-03-29T11:20:26.508 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:26.526+0000 7f483a7fc640 10 monclient: _check_auth_tickets 2024-03-29T11:20:26.508 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:26.526+0000 7f483a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:20:26.508 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:26.526+0000 7f484bfa8640 5 monclient: authenticate success, global_id 4268 2024-03-29T11:20:26.508 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:26.526+0000 7f484bfa8640 10 monclient: _renew_subs 2024-03-29T11:20:26.508 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:26.526+0000 7f484bfa8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-29T11:20:26.508 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:26.526+0000 7f484bfa8640 1 -- 172.21.15.84:0/4014126582 --> v1:172.21.15.84:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f48441d64a0 con 0x7f48440b0770 2024-03-29T11:20:26.508 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:26.526+0000 7f484bfa8640 10 client.4268.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:20:26.508 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:26.526+0000 7f484bfa8640 10 monclient: _renew_subs 2024-03-29T11:20:26.508 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:26.526+0000 7f484bfa8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-29T11:20:26.508 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:26.526+0000 7f484bfa8640 1 -- 172.21.15.84:0/4014126582 --> v1:172.21.15.84:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f4844148480 con 0x7f48440b0770 2024-03-29T11:20:26.508 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:26.526+0000 7f484bfa8640 1 librados: init done 2024-03-29T11:20:26.508 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:26.527+0000 7f483a7fc640 1 -- 172.21.15.84:0/4014126582 <== mon.2 v1:172.21.15.84:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4840002070 con 0x7f48440b0770 2024-03-29T11:20:26.509 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:26.527+0000 7f483a7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:20:26.509 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:26.527+0000 7f483b7fe640 4 set_mon_vals no callback set 2024-03-29T11:20:26.509 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:26.527+0000 7f483a7fc640 1 -- 172.21.15.84:0/4014126582 <== mon.2 v1:172.21.15.84:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4236868694 0 0) 0x7f4840003b10 con 0x7f48440b0770 2024-03-29T11:20:26.509 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:26.527+0000 7f483a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:26.509 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:26.527+0000 7f483a7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:20:26.509 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:26.527+0000 7f483a7fc640 10 monclient: dump: 2024-03-29T11:20:26.509 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T11:20:26.509 INFO:teuthology.misc.health.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:20:26.509 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T11:18:53.864025+0000 2024-03-29T11:20:26.509 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T11:18:53.864025+0000 2024-03-29T11:20:26.509 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:26.509 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:20:26.509 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T11:20:26.509 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T11:20:26.509 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T11:20:26.510 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:20:26.510 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:26.528+0000 7f4827fff640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:20:26.510 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:26.528+0000 7f4827fff640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:20:26.510 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:26.528+0000 7f4827fff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-29T11:20:26.510 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:26.528+0000 7f4827fff640 1 -- 172.21.15.84:0/4014126582 --> v1:172.21.15.84:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4820005080 con 0x7f48440b0770 2024-03-29T11:20:26.510 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:26.529+0000 7f483a7fc640 1 -- 172.21.15.84:0/4014126582 <== mon.2 v1:172.21.15.84:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4033942909 0 0) 0x7f4840019ae0 con 0x7f48440b0770 2024-03-29T11:20:26.510 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:26.529+0000 7f483a7fc640 1 -- 172.21.15.84:0/4014126582 <== mon.2 v1:172.21.15.84:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2808139587 0 0) 0x7f4840083990 con 0x7f48440b0770 2024-03-29T11:20:26.510 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:26.529+0000 7f483a7fc640 10 client.4268.objecter ms_dispatch 0x7f4844000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:20:26.510 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:26.529+0000 7f483a7fc640 3 client.4268.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:20:26.510 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:26.529+0000 7f483a7fc640 3 client.4268.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:20:26.510 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:26.529+0000 7f483a7fc640 20 client.4268.objecter dump_active .. 0 homeless 2024-03-29T11:20:26.510 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:26.529+0000 7f4849d44640 10 monclient: get_auth_request con 0x7f482806df80 auth_method 0 2024-03-29T11:20:26.510 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:26.530+0000 7f483a7fc640 10 client.4268.objecter ms_handle_connect 0x7f482806df80 2024-03-29T11:20:26.515 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:26.538+0000 7f483a7fc640 1 -- 172.21.15.84:0/4014126582 <== mon.2 v1:172.21.15.84:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f484004f1e0 con 0x7f48440b0770 2024-03-29T11:20:26.515 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:26.539+0000 7f483a7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:20:26.515 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:26.539+0000 7f483a7fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:20:26.837 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:26.861+0000 7f4827fff640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:20:26.838 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:26.861+0000 7f4827fff640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:20:26.838 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:26.861+0000 7f4827fff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-29T11:20:26.838 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:26.861+0000 7f4827fff640 1 -- 172.21.15.84:0/4014126582 --> v1:172.21.15.84:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4820005330 con 0x7f48440b0770 2024-03-29T11:20:26.838 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:26.862+0000 7f483a7fc640 1 -- 172.21.15.84:0/4014126582 <== mon.2 v1:172.21.15.84:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4209477549) 0x7f484004f1e0 con 0x7f48440b0770 2024-03-29T11:20:26.838 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:26.862+0000 7f483a7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:20:26.838 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:26.862+0000 7f483a7fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:20:26.840 INFO:teuthology.misc.health.smithi084.stdout: 2024-03-29T11:20:26.840 INFO:teuthology.misc.health.smithi084.stdout:{"epoch":10,"fsid":"2c3e002f-9eb0-4860-853b-4ef1ef562213","created":"2024-03-29T11:19:32.029169+0000","modified":"2024-03-29T11:19:34.200299+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:34.200299+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"5c39e276-2034-4716-bbc0-57a634eefa92","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"cd580e99-109f-4bf1-88c0-a5987bc8b9ed","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"11e60a5a-051d-453f-a190-66387cb35592","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"2c329fbe-bbc4-4030-bf87-d711cad04679","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f4213d4a-bd36-4433-91ae-d7e2b02481d3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"f0cf44ea-6055-48af-805b-aed858102ff9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"d895eaa0-cc0b-4ee4-9b5a-2dfcd89a60b7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"9bc5a7bc-8f52-4db9-96d6-de5e68d52eaa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:20:26.843 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:26.866+0000 7f4827fff640 10 librados: watch_flush enter 2024-03-29T11:20:26.843 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:26.867+0000 7f4827fff640 10 librados: watch_flush exit 2024-03-29T11:20:26.843 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:26.867+0000 7f4827fff640 20 client.4268.objecter shutdown clearing up homeless session... 2024-03-29T11:20:26.843 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:26.867+0000 7f4827fff640 10 client.4268.objecter successfully canceled tick 2024-03-29T11:20:26.844 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:26.867+0000 7f4827fff640 1 -- 172.21.15.84:0/4014126582 >> v1:172.21.15.179:6816/2981134980 conn(0x7f482806df80 legacy=0x7f4828070440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:26.844 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:26.867+0000 7f4827fff640 10 monclient: shutdown 2024-03-29T11:20:26.844 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:26.867+0000 7f4827fff640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:20:26.844 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:26.867+0000 7f4827fff640 1 -- 172.21.15.84:0/4014126582 >> v1:172.21.15.84:6790/0 conn(0x7f48440b0770 legacy=0x7f48441d6aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:26.844 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:26.867+0000 7f4827fff640 1 -- 172.21.15.84:0/4014126582 shutdown_connections 2024-03-29T11:20:26.844 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:26.867+0000 7f4827fff640 1 -- 172.21.15.84:0/4014126582 >> 172.21.15.84:0/4014126582 conn(0x7f484406b660 msgr2=0x7f48440abe50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:20:26.846 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:26.867+0000 7f4827fff640 1 -- 172.21.15.84:0/4014126582 shutdown_connections 2024-03-29T11:20:26.846 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:26.867+0000 7f4827fff640 1 -- 172.21.15.84:0/4014126582 wait complete. 2024-03-29T11:20:26.846 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:26.869+0000 7f4827fff640 1 librados: shutdown 2024-03-29T11:20:26.862 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:20:33.669 DEBUG:teuthology.orchestra.run.smithi084:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:20:33.820 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:33.843+0000 7fb12360d640 10 monclient: get_monmap_and_config 2024-03-29T11:20:33.821 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:33.843+0000 7fb12360d640 10 monclient: build_initial_monmap 2024-03-29T11:20:33.821 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:33.843+0000 7fb12360d640 10 monclient: monmap: 2024-03-29T11:20:33.821 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-29T11:20:33.821 INFO:teuthology.misc.health.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:20:33.821 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T11:20:33.844706+0000 2024-03-29T11:20:33.821 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T11:20:33.844706+0000 2024-03-29T11:20:33.821 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-29T11:20:33.821 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:20:33.821 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-29T11:20:33.821 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.noname-c 2024-03-29T11:20:33.821 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-b 2024-03-29T11:20:33.821 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:20:33.822 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:33.844+0000 7fb12360d640 1 Processor -- start 2024-03-29T11:20:33.822 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:33.844+0000 7fb12360d640 1 -- start start 2024-03-29T11:20:33.822 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:33.844+0000 7fb12360d640 10 monclient: init 2024-03-29T11:20:33.823 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:33.845+0000 7fb12360d640 10 monclient: _reopen_session rank -1 2024-03-29T11:20:33.823 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:33.845+0000 7fb12360d640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:20:33.823 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:33.845+0000 7fb12360d640 10 monclient(hunting): picked mon.noname-a con 0x7fb11c0a9750 addr v1:172.21.15.84:6789/0 2024-03-29T11:20:33.823 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:33.845+0000 7fb12360d640 10 monclient(hunting): picked mon.noname-c con 0x7fb11c13e1e0 addr v1:172.21.15.179:6789/0 2024-03-29T11:20:33.823 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:33.845+0000 7fb12360d640 10 monclient(hunting): picked mon.noname-b con 0x7fb11c071c30 addr v1:172.21.15.84:6790/0 2024-03-29T11:20:33.823 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:33.845+0000 7fb12360d640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb11c0ab300 con 0x7fb11c0a9750 2024-03-29T11:20:33.823 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:33.845+0000 7fb12360d640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb11c0ac2f0 con 0x7fb11c13e1e0 2024-03-29T11:20:33.823 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:33.845+0000 7fb12360d640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb11c0ad8d0 con 0x7fb11c071c30 2024-03-29T11:20:33.823 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:33.845+0000 7fb12360d640 10 monclient(hunting): _renew_subs 2024-03-29T11:20:33.823 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:33.845+0000 7fb12360d640 10 monclient(hunting): authenticate will time out at 895.139883s 2024-03-29T11:20:33.826 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:33.849+0000 7fb120ba8640 1 --1- >> v1:172.21.15.179:6789/0 conn(0x7fb11c13e1e0 0x7fb11c0ac710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.179:6789/0 says I am v1:172.21.15.84:43933/0 (socket says 172.21.15.84:43933) 2024-03-29T11:20:33.826 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:33.849+0000 7fb120ba8640 1 -- 172.21.15.84:0/1457891625 learned_addr learned my addr 172.21.15.84:0/1457891625 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-29T11:20:33.828 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:33.851+0000 7fb113fff640 1 -- 172.21.15.84:0/1457891625 <== mon.0 v1:172.21.15.84:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2903771791 0 0) 0x7fb11c0ab300 con 0x7fb11c0a9750 2024-03-29T11:20:33.828 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:33.851+0000 7fb113fff640 10 monclient(hunting): _negotiate 2024-03-29T11:20:33.828 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:33.851+0000 7fb113fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:33.828 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:33.852+0000 7fb113fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:33.828 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:33.852+0000 7fb113fff640 10 monclient(hunting): my global_id is 4275 2024-03-29T11:20:33.829 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:33.852+0000 7fb113fff640 1 -- 172.21.15.84:0/1457891625 --> v1:172.21.15.84:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb108003520 con 0x7fb11c0a9750 2024-03-29T11:20:33.829 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:33.852+0000 7fb113fff640 1 -- 172.21.15.84:0/1457891625 <== mon.1 v1:172.21.15.179:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 520139410 0 0) 0x7fb11c0ac2f0 con 0x7fb11c13e1e0 2024-03-29T11:20:33.829 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:33.852+0000 7fb113fff640 10 monclient(hunting): _negotiate 2024-03-29T11:20:33.829 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:33.852+0000 7fb113fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:33.829 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:33.852+0000 7fb113fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:33.829 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:33.852+0000 7fb113fff640 10 monclient(hunting): my global_id is 4246 2024-03-29T11:20:33.829 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:33.852+0000 7fb113fff640 1 -- 172.21.15.84:0/1457891625 --> v1:172.21.15.179:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb11c0ab300 con 0x7fb11c13e1e0 2024-03-29T11:20:33.829 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:33.852+0000 7fb113fff640 1 -- 172.21.15.84:0/1457891625 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2854167853 0 0) 0x7fb11c0ad8d0 con 0x7fb11c071c30 2024-03-29T11:20:33.829 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:33.852+0000 7fb113fff640 10 monclient(hunting): _negotiate 2024-03-29T11:20:33.829 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:33.852+0000 7fb113fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:33.829 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:33.852+0000 7fb113fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:33.829 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:33.852+0000 7fb113fff640 10 monclient(hunting): my global_id is 4271 2024-03-29T11:20:33.829 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:33.852+0000 7fb113fff640 1 -- 172.21.15.84:0/1457891625 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb11c0ac2f0 con 0x7fb11c071c30 2024-03-29T11:20:33.830 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:33.852+0000 7fb113fff640 1 -- 172.21.15.84:0/1457891625 <== mon.1 v1:172.21.15.179:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2164470879 0 0) 0x7fb11c0ab300 con 0x7fb11c13e1e0 2024-03-29T11:20:33.830 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:33.852+0000 7fb113fff640 1 -- 172.21.15.84:0/1457891625 >> v1:172.21.15.84:6790/0 conn(0x7fb11c071c30 legacy=0x7fb11c0ace20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:33.830 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:33.852+0000 7fb113fff640 1 -- 172.21.15.84:0/1457891625 >> v1:172.21.15.84:6789/0 conn(0x7fb11c0a9750 legacy=0x7fb11c13dad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:33.831 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:33.852+0000 7fb113fff640 10 monclient: _finish_hunting 0 2024-03-29T11:20:33.831 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:33.852+0000 7fb113fff640 1 monclient: found mon.noname-c 2024-03-29T11:20:33.831 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:33.852+0000 7fb113fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:20:33.831 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:33.852+0000 7fb113fff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.179:6789/0 2024-03-29T11:20:33.831 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:33.852+0000 7fb113fff640 1 -- 172.21.15.84:0/1457891625 --> v1:172.21.15.179:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb11c0ada60 con 0x7fb11c13e1e0 2024-03-29T11:20:33.831 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:33.852+0000 7fb113fff640 10 monclient: _finish_auth 0 2024-03-29T11:20:33.831 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:33.852+0000 7fb113fff640 10 monclient: _check_auth_tickets 2024-03-29T11:20:33.831 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:33.852+0000 7fb113fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:20:33.831 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:33.852+0000 7fb12360d640 5 monclient: authenticate success, global_id 4246 2024-03-29T11:20:33.831 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:33.852+0000 7fb12360d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:20:33.831 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:33.852+0000 7fb113fff640 1 -- 172.21.15.84:0/1457891625 <== mon.1 v1:172.21.15.179:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4236868694 0 0) 0x7fb10c0015f0 con 0x7fb11c13e1e0 2024-03-29T11:20:33.831 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:33.852+0000 7fb113fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:33.831 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:33.852+0000 7fb113fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:20:33.832 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:33.852+0000 7fb113fff640 10 monclient: dump: 2024-03-29T11:20:33.832 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T11:20:33.832 INFO:teuthology.misc.health.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:20:33.832 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T11:18:53.864025+0000 2024-03-29T11:20:33.832 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T11:18:53.864025+0000 2024-03-29T11:20:33.832 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:33.832 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:20:33.832 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T11:20:33.832 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T11:20:33.832 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T11:20:33.832 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:20:33.832 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:33.852+0000 7fb12360d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:20:33.832 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:33.852+0000 7fb113fff640 1 -- 172.21.15.84:0/1457891625 <== mon.1 v1:172.21.15.179:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb10c001750 con 0x7fb11c13e1e0 2024-03-29T11:20:33.832 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:33.853+0000 7fb113fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:20:33.833 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:33.853+0000 7fb12360d640 10 monclient: get_monmap_and_config success 2024-03-29T11:20:33.833 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:33.853+0000 7fb12360d640 4 set_mon_vals no callback set 2024-03-29T11:20:33.833 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:33.853+0000 7fb12360d640 10 monclient: shutdown 2024-03-29T11:20:33.833 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:33.853+0000 7fb12360d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:20:33.833 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:33.853+0000 7fb12360d640 1 -- 172.21.15.84:0/1457891625 >> v1:172.21.15.179:6789/0 conn(0x7fb11c13e1e0 legacy=0x7fb11c0ac710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:33.833 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:33.854+0000 7fb12360d640 1 -- 172.21.15.84:0/1457891625 shutdown_connections 2024-03-29T11:20:33.833 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:33.854+0000 7fb12360d640 1 -- 172.21.15.84:0/1457891625 >> 172.21.15.84:0/1457891625 conn(0x7fb11c06b660 msgr2=0x7fb11c06ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:20:33.834 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:33.855+0000 7fb12360d640 1 -- 172.21.15.84:0/1457891625 shutdown_connections 2024-03-29T11:20:33.834 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:33.855+0000 7fb12360d640 1 -- 172.21.15.84:0/1457891625 wait complete. 2024-03-29T11:20:33.834 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:33.856+0000 7fb12360d640 10 monclient: build_initial_monmap 2024-03-29T11:20:33.834 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:33.856+0000 7fb12360d640 10 monclient: monmap: 2024-03-29T11:20:33.834 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-29T11:20:33.834 INFO:teuthology.misc.health.smithi084.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:20:33.835 INFO:teuthology.misc.health.smithi084.stderr:last_changed 0.000000 2024-03-29T11:20:33.835 INFO:teuthology.misc.health.smithi084.stderr:created 0.000000 2024-03-29T11:20:33.835 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-29T11:20:33.835 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:20:33.835 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-29T11:20:33.835 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.noname-b 2024-03-29T11:20:33.835 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-c 2024-03-29T11:20:33.835 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:20:33.835 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:33.856+0000 7fb12360d640 1 librados: starting msgr at 2024-03-29T11:20:33.835 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:33.856+0000 7fb12360d640 1 librados: starting objecter 2024-03-29T11:20:33.835 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:33.856+0000 7fb12360d640 1 Processor -- start 2024-03-29T11:20:33.839 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:33.857+0000 7fb12360d640 1 -- start start 2024-03-29T11:20:33.839 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:33.857+0000 7fb12360d640 1 librados: setting wanted keys 2024-03-29T11:20:33.839 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:33.857+0000 7fb12360d640 1 librados: calling monclient init 2024-03-29T11:20:33.839 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:33.857+0000 7fb12360d640 10 monclient: init 2024-03-29T11:20:33.839 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:33.857+0000 7fb12360d640 10 monclient: _reopen_session rank -1 2024-03-29T11:20:33.839 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:33.857+0000 7fb12360d640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:20:33.839 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:33.857+0000 7fb12360d640 10 monclient(hunting): picked mon.noname-a con 0x7fb11c071c30 addr v1:172.21.15.84:6789/0 2024-03-29T11:20:33.840 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:33.857+0000 7fb12360d640 10 monclient(hunting): picked mon.noname-b con 0x7fb11c0a9750 addr v1:172.21.15.179:6789/0 2024-03-29T11:20:33.840 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:33.857+0000 7fb12360d640 10 monclient(hunting): picked mon.noname-c con 0x7fb11c13e1e0 addr v1:172.21.15.84:6790/0 2024-03-29T11:20:33.840 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:33.857+0000 7fb12360d640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb11c06f1f0 con 0x7fb11c071c30 2024-03-29T11:20:33.840 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:33.857+0000 7fb12360d640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb11c06f460 con 0x7fb11c0a9750 2024-03-29T11:20:33.840 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:33.857+0000 7fb12360d640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb11c0b3a30 con 0x7fb11c13e1e0 2024-03-29T11:20:33.840 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:33.857+0000 7fb12360d640 10 monclient(hunting): _renew_subs 2024-03-29T11:20:33.840 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:33.857+0000 7fb12360d640 10 monclient(hunting): authenticate will time out at 895.152042s 2024-03-29T11:20:33.840 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:33.861+0000 7fb121baa640 1 --1- >> v1:172.21.15.84:6790/0 conn(0x7fb11c13e1e0 0x7fb11c1e6510 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.84:6790/0 says I am v1:172.21.15.84:37013/0 (socket says 172.21.15.84:37013) 2024-03-29T11:20:33.840 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:33.861+0000 7fb121baa640 1 -- 172.21.15.84:0/3221082934 learned_addr learned my addr 172.21.15.84:0/3221082934 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-29T11:20:33.840 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:33.861+0000 7fb111ffb640 10 client.?.objecter ms_handle_connect 0x7fb11c13e1e0 2024-03-29T11:20:33.841 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:33.861+0000 7fb111ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:20:33.842 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:33.861+0000 7fb111ffb640 1 -- 172.21.15.84:0/3221082934 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4025441263 0 0) 0x7fb11c0b3a30 con 0x7fb11c13e1e0 2024-03-29T11:20:33.842 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:33.861+0000 7fb111ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:20:33.842 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:33.861+0000 7fb111ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:33.842 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:33.861+0000 7fb111ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:33.842 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:33.861+0000 7fb111ffb640 10 monclient(hunting): my global_id is 4274 2024-03-29T11:20:33.842 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:33.861+0000 7fb111ffb640 1 -- 172.21.15.84:0/3221082934 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb100004eb0 con 0x7fb11c13e1e0 2024-03-29T11:20:33.842 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:33.862+0000 7fb111ffb640 1 -- 172.21.15.84:0/3221082934 <== mon.2 v1:172.21.15.84:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4243873166 0 0) 0x7fb100004eb0 con 0x7fb11c13e1e0 2024-03-29T11:20:33.842 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:33.862+0000 7fb111ffb640 1 -- 172.21.15.84:0/3221082934 --> v1:172.21.15.84:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb100005300 con 0x7fb11c13e1e0 2024-03-29T11:20:33.842 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:33.862+0000 7fb111ffb640 1 -- 172.21.15.84:0/3221082934 <== mon.2 v1:172.21.15.84:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4236868694 0 0) 0x7fb114003f10 con 0x7fb11c13e1e0 2024-03-29T11:20:33.842 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:33.862+0000 7fb111ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:33.842 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:33.862+0000 7fb111ffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:20:33.842 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:33.862+0000 7fb111ffb640 10 monclient(hunting): dump: 2024-03-29T11:20:33.842 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T11:20:33.842 INFO:teuthology.misc.health.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:20:33.843 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T11:18:53.864025+0000 2024-03-29T11:20:33.843 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T11:18:53.864025+0000 2024-03-29T11:20:33.843 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:33.843 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:20:33.843 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T11:20:33.843 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T11:20:33.843 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T11:20:33.843 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:20:33.843 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:33.862+0000 7fb111ffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:20:33.843 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:33.862+0000 7fb111ffb640 1 -- 172.21.15.84:0/3221082934 <== mon.2 v1:172.21.15.84:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 924632951 0 0) 0x7fb100005300 con 0x7fb11c13e1e0 2024-03-29T11:20:33.843 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:33.862+0000 7fb111ffb640 1 -- 172.21.15.84:0/3221082934 >> v1:172.21.15.179:6789/0 conn(0x7fb11c0a9750 legacy=0x7fb11c1df8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:33.843 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:33.862+0000 7fb111ffb640 1 -- 172.21.15.84:0/3221082934 >> v1:172.21.15.84:6789/0 conn(0x7fb11c071c30 legacy=0x7fb11c1df1d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:20:33.845 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:33.863+0000 7fb111ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:20:33.845 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:33.863+0000 7fb111ffb640 1 monclient: found mon.c 2024-03-29T11:20:33.845 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:33.863+0000 7fb111ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:20:33.845 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:33.863+0000 7fb111ffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-29T11:20:33.846 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:33.863+0000 7fb111ffb640 1 -- 172.21.15.84:0/3221082934 --> v1:172.21.15.84:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb11c06e1b0 con 0x7fb11c13e1e0 2024-03-29T11:20:33.846 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:33.863+0000 7fb111ffb640 10 monclient: _finish_auth 0 2024-03-29T11:20:33.846 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:33.864+0000 7fb111ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:20:33.846 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:33.864+0000 7fb111ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:20:33.846 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:33.864+0000 7fb12360d640 5 monclient: authenticate success, global_id 4274 2024-03-29T11:20:33.846 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:33.864+0000 7fb12360d640 10 monclient: _renew_subs 2024-03-29T11:20:33.846 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:33.864+0000 7fb12360d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-29T11:20:33.846 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:33.864+0000 7fb12360d640 1 -- 172.21.15.84:0/3221082934 --> v1:172.21.15.84:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb11c06de90 con 0x7fb11c13e1e0 2024-03-29T11:20:33.846 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:33.864+0000 7fb12360d640 10 client.4274.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:20:33.846 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:33.864+0000 7fb12360d640 10 monclient: _renew_subs 2024-03-29T11:20:33.846 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:33.864+0000 7fb12360d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-29T11:20:33.846 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:33.864+0000 7fb12360d640 1 -- 172.21.15.84:0/3221082934 --> v1:172.21.15.84:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb11c0b25a0 con 0x7fb11c13e1e0 2024-03-29T11:20:33.846 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:33.864+0000 7fb12360d640 1 librados: init done 2024-03-29T11:20:33.847 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:33.864+0000 7fb111ffb640 1 -- 172.21.15.84:0/3221082934 <== mon.2 v1:172.21.15.84:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb1140044c0 con 0x7fb11c13e1e0 2024-03-29T11:20:33.847 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:33.864+0000 7fb111ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:20:33.847 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:33.864+0000 7fb111ffb640 1 -- 172.21.15.84:0/3221082934 <== mon.2 v1:172.21.15.84:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4236868694 0 0) 0x7fb114003f10 con 0x7fb11c13e1e0 2024-03-29T11:20:33.847 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:33.864+0000 7fb111ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:33.847 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:33.864+0000 7fb1137fe640 4 set_mon_vals no callback set 2024-03-29T11:20:33.847 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:33.864+0000 7fb111ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:20:33.847 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:33.864+0000 7fb111ffb640 10 monclient: dump: 2024-03-29T11:20:33.847 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T11:20:33.847 INFO:teuthology.misc.health.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:20:33.847 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T11:18:53.864025+0000 2024-03-29T11:20:33.847 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T11:18:53.864025+0000 2024-03-29T11:20:33.847 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:33.847 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:20:33.847 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T11:20:33.847 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T11:20:33.847 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T11:20:33.847 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:20:33.848 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:33.868+0000 7fb111ffb640 1 -- 172.21.15.84:0/3221082934 <== mon.2 v1:172.21.15.84:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4033942909 0 0) 0x7fb114000fd0 con 0x7fb11c13e1e0 2024-03-29T11:20:33.848 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:33.868+0000 7fb12360d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:20:33.848 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:33.868+0000 7fb12360d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:20:33.848 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:33.868+0000 7fb12360d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-29T11:20:33.848 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:33.868+0000 7fb12360d640 1 -- 172.21.15.84:0/3221082934 --> v1:172.21.15.84:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb0fc005080 con 0x7fb11c13e1e0 2024-03-29T11:20:33.849 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:33.869+0000 7fb111ffb640 1 -- 172.21.15.84:0/3221082934 <== mon.2 v1:172.21.15.84:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2808139587 0 0) 0x7fb114084510 con 0x7fb11c13e1e0 2024-03-29T11:20:33.849 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:33.869+0000 7fb111ffb640 10 client.4274.objecter ms_dispatch 0x7fb11c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:20:33.849 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:33.869+0000 7fb111ffb640 3 client.4274.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:20:33.849 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:33.869+0000 7fb111ffb640 3 client.4274.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:20:33.849 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:33.870+0000 7fb111ffb640 20 client.4274.objecter dump_active .. 0 homeless 2024-03-29T11:20:33.849 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:33.870+0000 7fb1213a9640 10 monclient: get_auth_request con 0x7fb10006e130 auth_method 0 2024-03-29T11:20:33.851 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:33.873+0000 7fb111ffb640 10 client.4274.objecter ms_handle_connect 0x7fb10006e130 2024-03-29T11:20:33.855 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:33.879+0000 7fb111ffb640 1 -- 172.21.15.84:0/3221082934 <== mon.2 v1:172.21.15.84:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fb11404fe50 con 0x7fb11c13e1e0 2024-03-29T11:20:33.855 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:33.879+0000 7fb111ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:20:33.855 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:33.879+0000 7fb111ffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:20:34.180 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:34.203+0000 7fb12360d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:20:34.180 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:34.203+0000 7fb12360d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:20:34.180 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:34.203+0000 7fb12360d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-29T11:20:34.180 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:34.203+0000 7fb12360d640 1 -- 172.21.15.84:0/3221082934 --> v1:172.21.15.84:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb0fc005330 con 0x7fb11c13e1e0 2024-03-29T11:20:34.182 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:34.206+0000 7fb111ffb640 1 -- 172.21.15.84:0/3221082934 <== mon.2 v1:172.21.15.84:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4209477549) 0x7fb114003d70 con 0x7fb11c13e1e0 2024-03-29T11:20:34.182 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:34.206+0000 7fb111ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:20:34.182 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:34.206+0000 7fb111ffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:20:34.183 INFO:teuthology.misc.health.smithi084.stdout: 2024-03-29T11:20:34.184 INFO:teuthology.misc.health.smithi084.stdout:{"epoch":10,"fsid":"2c3e002f-9eb0-4860-853b-4ef1ef562213","created":"2024-03-29T11:19:32.029169+0000","modified":"2024-03-29T11:19:34.200299+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:34.200299+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"5c39e276-2034-4716-bbc0-57a634eefa92","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"cd580e99-109f-4bf1-88c0-a5987bc8b9ed","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"11e60a5a-051d-453f-a190-66387cb35592","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"2c329fbe-bbc4-4030-bf87-d711cad04679","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f4213d4a-bd36-4433-91ae-d7e2b02481d3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"f0cf44ea-6055-48af-805b-aed858102ff9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"d895eaa0-cc0b-4ee4-9b5a-2dfcd89a60b7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"9bc5a7bc-8f52-4db9-96d6-de5e68d52eaa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:20:34.186 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:34.209+0000 7fb12360d640 10 librados: watch_flush enter 2024-03-29T11:20:34.186 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:34.209+0000 7fb12360d640 10 librados: watch_flush exit 2024-03-29T11:20:34.186 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:34.209+0000 7fb12360d640 20 client.4274.objecter shutdown clearing up homeless session... 2024-03-29T11:20:34.186 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:34.209+0000 7fb12360d640 10 client.4274.objecter successfully canceled tick 2024-03-29T11:20:34.186 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:34.209+0000 7fb12360d640 1 -- 172.21.15.84:0/3221082934 >> v1:172.21.15.179:6816/2981134980 conn(0x7fb10006e130 legacy=0x7fb1000705f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:34.186 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:34.209+0000 7fb12360d640 10 monclient: shutdown 2024-03-29T11:20:34.186 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:34.209+0000 7fb12360d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:20:34.186 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:34.209+0000 7fb12360d640 1 -- 172.21.15.84:0/3221082934 >> v1:172.21.15.84:6790/0 conn(0x7fb11c13e1e0 legacy=0x7fb11c1e6510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:34.186 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:34.210+0000 7fb12360d640 1 -- 172.21.15.84:0/3221082934 shutdown_connections 2024-03-29T11:20:34.187 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:34.210+0000 7fb12360d640 1 -- 172.21.15.84:0/3221082934 >> 172.21.15.84:0/3221082934 conn(0x7fb11c06b660 msgr2=0x7fb11c06ba40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:20:34.188 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:34.211+0000 7fb12360d640 1 -- 172.21.15.84:0/3221082934 shutdown_connections 2024-03-29T11:20:34.188 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:34.212+0000 7fb12360d640 1 -- 172.21.15.84:0/3221082934 wait complete. 2024-03-29T11:20:34.188 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:34.212+0000 7fb12360d640 1 librados: shutdown 2024-03-29T11:20:34.207 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:20:41.015 DEBUG:teuthology.orchestra.run.smithi084:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:20:41.169 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:41.192+0000 7f1dd1801640 10 monclient: get_monmap_and_config 2024-03-29T11:20:41.169 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:41.193+0000 7f1dd1801640 10 monclient: build_initial_monmap 2024-03-29T11:20:41.169 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:41.193+0000 7f1dd1801640 10 monclient: monmap: 2024-03-29T11:20:41.169 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-29T11:20:41.169 INFO:teuthology.misc.health.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:20:41.170 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T11:20:41.193948+0000 2024-03-29T11:20:41.170 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T11:20:41.193948+0000 2024-03-29T11:20:41.170 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-29T11:20:41.170 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:20:41.170 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-29T11:20:41.170 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.noname-c 2024-03-29T11:20:41.170 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-b 2024-03-29T11:20:41.170 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:20:41.175 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:41.195+0000 7f1dd1801640 1 Processor -- start 2024-03-29T11:20:41.175 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:41.195+0000 7f1dd1801640 1 -- start start 2024-03-29T11:20:41.175 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:41.195+0000 7f1dd1801640 10 monclient: init 2024-03-29T11:20:41.175 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:41.196+0000 7f1dd1801640 10 monclient: _reopen_session rank -1 2024-03-29T11:20:41.175 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:41.196+0000 7f1dd1801640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:20:41.175 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:41.196+0000 7f1dd1801640 10 monclient(hunting): picked mon.noname-a con 0x7f1dcc0b2790 addr v1:172.21.15.84:6789/0 2024-03-29T11:20:41.175 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:41.196+0000 7f1dd1801640 10 monclient(hunting): picked mon.noname-c con 0x7f1dcc143c80 addr v1:172.21.15.179:6789/0 2024-03-29T11:20:41.175 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:41.196+0000 7f1dd1801640 10 monclient(hunting): picked mon.noname-b con 0x7f1dcc146e30 addr v1:172.21.15.84:6790/0 2024-03-29T11:20:41.175 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:41.196+0000 7f1dd1801640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1dcc146290 con 0x7f1dcc0b2790 2024-03-29T11:20:41.175 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:41.196+0000 7f1dd1801640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1dcc0b3970 con 0x7f1dcc143c80 2024-03-29T11:20:41.176 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:41.196+0000 7f1dd1801640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1dcc06ee80 con 0x7f1dcc146e30 2024-03-29T11:20:41.176 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:41.196+0000 7f1dd1801640 10 monclient(hunting): _renew_subs 2024-03-29T11:20:41.176 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:41.196+0000 7f1dd1801640 10 monclient(hunting): authenticate will time out at 902.490509s 2024-03-29T11:20:41.176 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:41.198+0000 7f1dcbfff640 1 --1- >> v1:172.21.15.84:6789/0 conn(0x7f1dcc0b2790 0x7f1dcc143570 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.84:6789/0 says I am v1:172.21.15.84:58757/0 (socket says 172.21.15.84:58757) 2024-03-29T11:20:41.176 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:41.198+0000 7f1dcbfff640 1 -- 172.21.15.84:0/2485624761 learned_addr learned my addr 172.21.15.84:0/2485624761 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-29T11:20:41.176 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:41.199+0000 7f1dcaffd640 1 -- 172.21.15.84:0/2485624761 <== mon.0 v1:172.21.15.84:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3128005440 0 0) 0x7f1dcc146290 con 0x7f1dcc0b2790 2024-03-29T11:20:41.176 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:41.199+0000 7f1dcaffd640 10 monclient(hunting): _negotiate 2024-03-29T11:20:41.176 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:41.199+0000 7f1dcaffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:41.176 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:41.199+0000 7f1dcaffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:41.176 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:41.199+0000 7f1dcaffd640 10 monclient(hunting): my global_id is 4278 2024-03-29T11:20:41.177 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:41.199+0000 7f1dcaffd640 1 -- 172.21.15.84:0/2485624761 --> v1:172.21.15.84:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1db4003520 con 0x7f1dcc0b2790 2024-03-29T11:20:41.179 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:41.203+0000 7f1dcaffd640 1 -- 172.21.15.84:0/2485624761 <== mon.0 v1:172.21.15.84:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3860190507 0 0) 0x7f1db4003520 con 0x7f1dcc0b2790 2024-03-29T11:20:41.179 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:41.203+0000 7f1dcaffd640 1 -- 172.21.15.84:0/2485624761 >> v1:172.21.15.84:6790/0 conn(0x7f1dcc146e30 legacy=0x7f1dcc151bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:20:41.179 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:41.203+0000 7f1dcaffd640 1 -- 172.21.15.84:0/2485624761 >> v1:172.21.15.179:6789/0 conn(0x7f1dcc143c80 legacy=0x7f1dcc146720 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:20:41.182 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:41.203+0000 7f1dcaffd640 10 monclient: _finish_hunting 0 2024-03-29T11:20:41.182 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:41.203+0000 7f1dcaffd640 1 monclient: found mon.noname-a 2024-03-29T11:20:41.182 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:41.203+0000 7f1dcaffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:20:41.182 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:41.203+0000 7f1dcaffd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.84:6789/0 2024-03-29T11:20:41.182 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:41.203+0000 7f1dcaffd640 1 -- 172.21.15.84:0/2485624761 --> v1:172.21.15.84:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1dcc06f160 con 0x7f1dcc0b2790 2024-03-29T11:20:41.182 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:41.204+0000 7f1dcaffd640 10 monclient: _finish_auth 0 2024-03-29T11:20:41.182 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:41.204+0000 7f1dcaffd640 10 monclient: _check_auth_tickets 2024-03-29T11:20:41.182 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:41.204+0000 7f1dcaffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:20:41.182 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:41.204+0000 7f1dd1801640 5 monclient: authenticate success, global_id 4278 2024-03-29T11:20:41.182 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:41.204+0000 7f1dd1801640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:20:41.182 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:41.204+0000 7f1dcaffd640 1 -- 172.21.15.84:0/2485624761 <== mon.0 v1:172.21.15.84:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4236868694 0 0) 0x7f1dc4001a10 con 0x7f1dcc0b2790 2024-03-29T11:20:41.183 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:41.204+0000 7f1dcaffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:41.183 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:41.204+0000 7f1dcaffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:20:41.183 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:41.204+0000 7f1dcaffd640 10 monclient: dump: 2024-03-29T11:20:41.183 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T11:20:41.183 INFO:teuthology.misc.health.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:20:41.183 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T11:18:53.864025+0000 2024-03-29T11:20:41.183 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T11:18:53.864025+0000 2024-03-29T11:20:41.183 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:41.183 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:20:41.183 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T11:20:41.183 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T11:20:41.183 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T11:20:41.183 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:20:41.184 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:41.204+0000 7f1dd1801640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:20:41.184 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:41.204+0000 7f1dcaffd640 1 -- 172.21.15.84:0/2485624761 <== mon.0 v1:172.21.15.84:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1dc4001ca0 con 0x7f1dcc0b2790 2024-03-29T11:20:41.184 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:41.204+0000 7f1dcaffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:20:41.184 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:41.204+0000 7f1dd1801640 10 monclient: get_monmap_and_config success 2024-03-29T11:20:41.184 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:41.204+0000 7f1dd1801640 4 set_mon_vals no callback set 2024-03-29T11:20:41.184 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:41.205+0000 7f1dd1801640 10 monclient: shutdown 2024-03-29T11:20:41.184 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:41.205+0000 7f1dd1801640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:20:41.184 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:41.205+0000 7f1dd1801640 1 -- 172.21.15.84:0/2485624761 >> v1:172.21.15.84:6789/0 conn(0x7f1dcc0b2790 legacy=0x7f1dcc143570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:41.185 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:41.205+0000 7f1dd1801640 1 -- 172.21.15.84:0/2485624761 shutdown_connections 2024-03-29T11:20:41.185 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:41.206+0000 7f1dd1801640 1 -- 172.21.15.84:0/2485624761 >> 172.21.15.84:0/2485624761 conn(0x7f1dcc06d070 msgr2=0x7f1dcc066a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:20:41.185 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:41.208+0000 7f1dd1801640 1 -- 172.21.15.84:0/2485624761 shutdown_connections 2024-03-29T11:20:41.185 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:41.209+0000 7f1dd1801640 1 -- 172.21.15.84:0/2485624761 wait complete. 2024-03-29T11:20:41.186 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:41.210+0000 7f1dd1801640 10 monclient: build_initial_monmap 2024-03-29T11:20:41.190 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:41.210+0000 7f1dd1801640 10 monclient: monmap: 2024-03-29T11:20:41.190 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-29T11:20:41.190 INFO:teuthology.misc.health.smithi084.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:20:41.190 INFO:teuthology.misc.health.smithi084.stderr:last_changed 0.000000 2024-03-29T11:20:41.190 INFO:teuthology.misc.health.smithi084.stderr:created 0.000000 2024-03-29T11:20:41.191 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-29T11:20:41.191 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:20:41.191 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-29T11:20:41.191 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.noname-b 2024-03-29T11:20:41.191 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-c 2024-03-29T11:20:41.191 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:20:41.192 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:41.210+0000 7f1dd1801640 1 librados: starting msgr at 2024-03-29T11:20:41.192 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:41.210+0000 7f1dd1801640 1 librados: starting objecter 2024-03-29T11:20:41.192 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:41.210+0000 7f1dd1801640 1 Processor -- start 2024-03-29T11:20:41.192 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:41.211+0000 7f1dd1801640 1 -- start start 2024-03-29T11:20:41.192 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:41.211+0000 7f1dd1801640 1 librados: setting wanted keys 2024-03-29T11:20:41.193 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:41.211+0000 7f1dd1801640 1 librados: calling monclient init 2024-03-29T11:20:41.193 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:41.211+0000 7f1dd1801640 10 monclient: init 2024-03-29T11:20:41.193 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:41.212+0000 7f1dd1801640 10 monclient: _reopen_session rank -1 2024-03-29T11:20:41.193 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:41.212+0000 7f1dd1801640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:20:41.193 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:41.212+0000 7f1dd1801640 10 monclient(hunting): picked mon.noname-b con 0x7f1dcc0b2790 addr v1:172.21.15.179:6789/0 2024-03-29T11:20:41.193 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:41.212+0000 7f1dd1801640 10 monclient(hunting): picked mon.noname-a con 0x7f1dcc143c80 addr v1:172.21.15.84:6789/0 2024-03-29T11:20:41.193 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:41.212+0000 7f1dd1801640 10 monclient(hunting): picked mon.noname-c con 0x7f1dcc146e30 addr v1:172.21.15.84:6790/0 2024-03-29T11:20:41.193 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:41.212+0000 7f1dd1801640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1dcc0b7890 con 0x7f1dcc143c80 2024-03-29T11:20:41.193 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:41.212+0000 7f1dd1801640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1dcc0b6c60 con 0x7f1dcc0b2790 2024-03-29T11:20:41.193 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:41.212+0000 7f1dd1801640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1dcc144340 con 0x7f1dcc146e30 2024-03-29T11:20:41.193 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:41.212+0000 7f1dd1801640 10 monclient(hunting): _renew_subs 2024-03-29T11:20:41.193 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:41.212+0000 7f1dd1801640 10 monclient(hunting): authenticate will time out at 902.506455s 2024-03-29T11:20:41.193 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:41.215+0000 7f1dcbfff640 1 --1- >> v1:172.21.15.179:6789/0 conn(0x7f1dcc0b2790 0x7f1dcc0b7180 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.179:6789/0 says I am v1:172.21.15.84:36657/0 (socket says 172.21.15.84:36657) 2024-03-29T11:20:41.193 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:41.215+0000 7f1dcbfff640 1 -- 172.21.15.84:0/1841951547 learned_addr learned my addr 172.21.15.84:0/1841951547 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-29T11:20:41.194 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:41.215+0000 7f1dc8ff9640 10 client.?.objecter ms_handle_connect 0x7f1dcc0b2790 2024-03-29T11:20:41.194 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:41.215+0000 7f1dc8ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:20:41.196 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:41.218+0000 7f1dc8ff9640 10 client.?.objecter ms_handle_connect 0x7f1dcc146e30 2024-03-29T11:20:41.196 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:41.218+0000 7f1dc8ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:20:41.196 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:41.219+0000 7f1dc8ff9640 1 -- 172.21.15.84:0/1841951547 <== mon.1 v1:172.21.15.179:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3938736905 0 0) 0x7f1dcc0b6c60 con 0x7f1dcc0b2790 2024-03-29T11:20:41.196 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:41.219+0000 7f1dc8ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:20:41.196 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:41.219+0000 7f1dc8ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:41.196 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:41.219+0000 7f1dc8ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:41.196 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:41.219+0000 7f1dc8ff9640 10 monclient(hunting): my global_id is 4249 2024-03-29T11:20:41.197 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:41.219+0000 7f1dc8ff9640 1 -- 172.21.15.84:0/1841951547 --> v1:172.21.15.179:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1db8004eb0 con 0x7f1dcc0b2790 2024-03-29T11:20:41.197 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:41.219+0000 7f1dc8ff9640 10 client.?.objecter ms_handle_connect 0x7f1dcc143c80 2024-03-29T11:20:41.198 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:41.219+0000 7f1dc8ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:20:41.198 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:41.219+0000 7f1dc8ff9640 1 -- 172.21.15.84:0/1841951547 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4243753430 0 0) 0x7f1dcc144340 con 0x7f1dcc146e30 2024-03-29T11:20:41.198 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:41.219+0000 7f1dc8ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:20:41.204 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:41.219+0000 7f1dc8ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:41.204 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:41.219+0000 7f1dc8ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:41.204 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:41.219+0000 7f1dc8ff9640 10 monclient(hunting): my global_id is 4277 2024-03-29T11:20:41.205 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:41.219+0000 7f1dc8ff9640 1 -- 172.21.15.84:0/1841951547 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1dcc0b6c60 con 0x7f1dcc146e30 2024-03-29T11:20:41.205 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:41.219+0000 7f1dc8ff9640 1 -- 172.21.15.84:0/1841951547 <== mon.1 v1:172.21.15.179:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3398322467 0 0) 0x7f1db8004eb0 con 0x7f1dcc0b2790 2024-03-29T11:20:41.205 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:41.219+0000 7f1dc8ff9640 1 -- 172.21.15.84:0/1841951547 --> v1:172.21.15.179:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1dcc144340 con 0x7f1dcc0b2790 2024-03-29T11:20:41.205 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:41.219+0000 7f1dc8ff9640 1 -- 172.21.15.84:0/1841951547 <== mon.1 v1:172.21.15.179:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4236868694 0 0) 0x7f1dc4004e60 con 0x7f1dcc0b2790 2024-03-29T11:20:41.205 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:41.219+0000 7f1dc8ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:41.205 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:41.219+0000 7f1dc8ff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:20:41.205 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:41.219+0000 7f1dc8ff9640 10 monclient(hunting): dump: 2024-03-29T11:20:41.205 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T11:20:41.205 INFO:teuthology.misc.health.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:20:41.205 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T11:18:53.864025+0000 2024-03-29T11:20:41.205 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T11:18:53.864025+0000 2024-03-29T11:20:41.205 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:41.205 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:20:41.205 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T11:20:41.206 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T11:20:41.206 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T11:20:41.206 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:20:41.206 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:41.219+0000 7f1dc8ff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:20:41.206 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:41.219+0000 7f1dc8ff9640 1 -- 172.21.15.84:0/1841951547 <== mon.2 v1:172.21.15.84:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2567395079 0 0) 0x7f1dcc0b6c60 con 0x7f1dcc146e30 2024-03-29T11:20:41.206 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:41.219+0000 7f1dc8ff9640 1 -- 172.21.15.84:0/1841951547 --> v1:172.21.15.84:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1db8004eb0 con 0x7f1dcc146e30 2024-03-29T11:20:41.206 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:41.219+0000 7f1dc8ff9640 1 -- 172.21.15.84:0/1841951547 <== mon.2 v1:172.21.15.84:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4236868694 0 0) 0x7f1dbc0016f0 con 0x7f1dcc146e30 2024-03-29T11:20:41.206 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:41.219+0000 7f1dc8ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:41.206 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:41.219+0000 7f1dc8ff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:20:41.206 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:41.219+0000 7f1dc8ff9640 10 monclient(hunting): dump: 2024-03-29T11:20:41.206 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T11:20:41.206 INFO:teuthology.misc.health.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:20:41.206 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T11:18:53.864025+0000 2024-03-29T11:20:41.207 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T11:18:53.864025+0000 2024-03-29T11:20:41.207 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:41.207 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:20:41.207 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T11:20:41.207 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T11:20:41.207 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T11:20:41.207 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:20:41.207 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:41.220+0000 7f1dc8ff9640 1 -- 172.21.15.84:0/1841951547 <== mon.2 v1:172.21.15.84:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2979251224 0 0) 0x7f1db8004eb0 con 0x7f1dcc146e30 2024-03-29T11:20:41.207 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:41.220+0000 7f1dc8ff9640 1 -- 172.21.15.84:0/1841951547 >> v1:172.21.15.179:6789/0 conn(0x7f1dcc0b2790 legacy=0x7f1dcc0b7180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:41.207 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:41.220+0000 7f1dc8ff9640 1 -- 172.21.15.84:0/1841951547 >> v1:172.21.15.84:6789/0 conn(0x7f1dcc143c80 legacy=0x7f1dcc1d89d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:41.207 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:41.220+0000 7f1dc8ff9640 10 monclient: _finish_hunting 0 2024-03-29T11:20:41.207 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:41.220+0000 7f1dc8ff9640 1 monclient: found mon.c 2024-03-29T11:20:41.207 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:41.220+0000 7f1dc8ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:20:41.207 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:41.220+0000 7f1dc8ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-29T11:20:41.207 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:41.220+0000 7f1dc8ff9640 1 -- 172.21.15.84:0/1841951547 --> v1:172.21.15.84:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1dcc1d90e0 con 0x7f1dcc146e30 2024-03-29T11:20:41.208 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:41.220+0000 7f1dc8ff9640 10 monclient: _finish_auth 0 2024-03-29T11:20:41.208 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:41.220+0000 7f1dc8ff9640 10 monclient: _check_auth_tickets 2024-03-29T11:20:41.208 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:41.220+0000 7f1dc8ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:20:41.208 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:41.220+0000 7f1dd1801640 5 monclient: authenticate success, global_id 4277 2024-03-29T11:20:41.208 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:41.220+0000 7f1dd1801640 10 monclient: _renew_subs 2024-03-29T11:20:41.208 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:41.220+0000 7f1dd1801640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-29T11:20:41.208 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:41.220+0000 7f1dd1801640 1 -- 172.21.15.84:0/1841951547 --> v1:172.21.15.84:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1dcc145370 con 0x7f1dcc146e30 2024-03-29T11:20:41.208 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:41.220+0000 7f1dd1801640 10 client.4277.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:20:41.208 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:41.220+0000 7f1dd1801640 10 monclient: _renew_subs 2024-03-29T11:20:41.208 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:41.220+0000 7f1dd1801640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-29T11:20:41.208 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:41.220+0000 7f1dd1801640 1 -- 172.21.15.84:0/1841951547 --> v1:172.21.15.84:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f1dcc1460b0 con 0x7f1dcc146e30 2024-03-29T11:20:41.208 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:41.220+0000 7f1dd1801640 1 librados: init done 2024-03-29T11:20:41.208 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:41.221+0000 7f1dc8ff9640 1 -- 172.21.15.84:0/1841951547 <== mon.2 v1:172.21.15.84:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1dbc001be0 con 0x7f1dcc146e30 2024-03-29T11:20:41.208 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:41.221+0000 7f1dc8ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:20:41.208 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:41.221+0000 7f1dc8ff9640 1 -- 172.21.15.84:0/1841951547 <== mon.2 v1:172.21.15.84:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4236868694 0 0) 0x7f1dbc003790 con 0x7f1dcc146e30 2024-03-29T11:20:41.208 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:41.221+0000 7f1dc8ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:41.209 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:41.221+0000 7f1dc8ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:20:41.209 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:41.221+0000 7f1dc8ff9640 10 monclient: dump: 2024-03-29T11:20:41.209 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T11:20:41.209 INFO:teuthology.misc.health.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:20:41.209 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T11:18:53.864025+0000 2024-03-29T11:20:41.209 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T11:18:53.864025+0000 2024-03-29T11:20:41.209 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:41.209 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:20:41.209 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T11:20:41.209 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T11:20:41.209 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T11:20:41.209 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:20:41.209 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:41.221+0000 7f1dca7fc640 4 set_mon_vals no callback set 2024-03-29T11:20:41.209 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:41.221+0000 7f1dd1801640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:20:41.209 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:41.222+0000 7f1dd1801640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:20:41.209 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:41.222+0000 7f1dd1801640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-29T11:20:41.210 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:41.222+0000 7f1dd1801640 1 -- 172.21.15.84:0/1841951547 --> v1:172.21.15.84:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1dcc0b6df0 con 0x7f1dcc146e30 2024-03-29T11:20:41.210 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:41.224+0000 7f1dc8ff9640 1 -- 172.21.15.84:0/1841951547 <== mon.2 v1:172.21.15.84:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4033942909 0 0) 0x7f1dbc019760 con 0x7f1dcc146e30 2024-03-29T11:20:41.210 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:41.224+0000 7f1dc8ff9640 1 -- 172.21.15.84:0/1841951547 <== mon.2 v1:172.21.15.84:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2808139587 0 0) 0x7f1dbc083620 con 0x7f1dcc146e30 2024-03-29T11:20:41.210 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:41.224+0000 7f1dc8ff9640 10 client.4277.objecter ms_dispatch 0x7f1dcc000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:20:41.210 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:41.224+0000 7f1dc8ff9640 3 client.4277.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:20:41.210 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:41.224+0000 7f1dc8ff9640 3 client.4277.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:20:41.210 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:41.225+0000 7f1dc8ff9640 20 client.4277.objecter dump_active .. 0 homeless 2024-03-29T11:20:41.210 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:41.225+0000 7f1dcbfff640 10 monclient: get_auth_request con 0x7f1db806dec0 auth_method 0 2024-03-29T11:20:41.210 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:41.225+0000 7f1dc8ff9640 10 client.4277.objecter ms_handle_connect 0x7f1db806dec0 2024-03-29T11:20:41.211 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:41.234+0000 7f1dc8ff9640 1 -- 172.21.15.84:0/1841951547 <== mon.2 v1:172.21.15.84:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f1dbc04ef00 con 0x7f1dcc146e30 2024-03-29T11:20:41.211 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:41.234+0000 7f1dc8ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:20:41.211 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:41.234+0000 7f1dc8ff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:20:41.554 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:41.577+0000 7f1dd1801640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:20:41.554 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:41.577+0000 7f1dd1801640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:20:41.554 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:41.577+0000 7f1dd1801640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-29T11:20:41.554 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:41.577+0000 7f1dd1801640 1 -- 172.21.15.84:0/1841951547 --> v1:172.21.15.84:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1dcc13c6f0 con 0x7f1dcc146e30 2024-03-29T11:20:41.554 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:41.578+0000 7f1dc8ff9640 1 -- 172.21.15.84:0/1841951547 <== mon.2 v1:172.21.15.84:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4209477549) 0x7f1dbc050e30 con 0x7f1dcc146e30 2024-03-29T11:20:41.554 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:41.578+0000 7f1dc8ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:20:41.555 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:41.578+0000 7f1dc8ff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:20:41.557 INFO:teuthology.misc.health.smithi084.stdout: 2024-03-29T11:20:41.557 INFO:teuthology.misc.health.smithi084.stdout:{"epoch":10,"fsid":"2c3e002f-9eb0-4860-853b-4ef1ef562213","created":"2024-03-29T11:19:32.029169+0000","modified":"2024-03-29T11:19:34.200299+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:34.200299+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"5c39e276-2034-4716-bbc0-57a634eefa92","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"cd580e99-109f-4bf1-88c0-a5987bc8b9ed","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"11e60a5a-051d-453f-a190-66387cb35592","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"2c329fbe-bbc4-4030-bf87-d711cad04679","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f4213d4a-bd36-4433-91ae-d7e2b02481d3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"f0cf44ea-6055-48af-805b-aed858102ff9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"d895eaa0-cc0b-4ee4-9b5a-2dfcd89a60b7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"9bc5a7bc-8f52-4db9-96d6-de5e68d52eaa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:20:41.562 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:41.585+0000 7f1dd1801640 10 librados: watch_flush enter 2024-03-29T11:20:41.563 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:41.586+0000 7f1dd1801640 10 librados: watch_flush exit 2024-03-29T11:20:41.563 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:41.586+0000 7f1dd1801640 20 client.4277.objecter shutdown clearing up homeless session... 2024-03-29T11:20:41.563 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:41.586+0000 7f1dd1801640 10 client.4277.objecter successfully canceled tick 2024-03-29T11:20:41.563 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:41.586+0000 7f1dd1801640 1 -- 172.21.15.84:0/1841951547 >> v1:172.21.15.179:6816/2981134980 conn(0x7f1db806dec0 legacy=0x7f1db8070380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:41.563 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:41.587+0000 7f1dd1801640 10 monclient: shutdown 2024-03-29T11:20:41.563 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:41.587+0000 7f1dd1801640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:20:41.563 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:41.587+0000 7f1dd1801640 1 -- 172.21.15.84:0/1841951547 >> v1:172.21.15.84:6790/0 conn(0x7f1dcc146e30 legacy=0x7f1dcc1457c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:41.564 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:41.587+0000 7f1dd1801640 1 -- 172.21.15.84:0/1841951547 shutdown_connections 2024-03-29T11:20:41.564 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:41.587+0000 7f1dd1801640 1 -- 172.21.15.84:0/1841951547 >> 172.21.15.84:0/1841951547 conn(0x7f1dcc06d070 msgr2=0x7f1dcc0b3b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:20:41.568 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:41.591+0000 7f1dd1801640 1 -- 172.21.15.84:0/1841951547 shutdown_connections 2024-03-29T11:20:41.569 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:41.592+0000 7f1dd1801640 1 -- 172.21.15.84:0/1841951547 wait complete. 2024-03-29T11:20:41.569 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:41.593+0000 7f1dd1801640 1 librados: shutdown 2024-03-29T11:20:41.588 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:20:48.395 DEBUG:teuthology.orchestra.run.smithi084:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:20:48.564 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:48.570+0000 7fc5b4a86640 10 monclient: get_monmap_and_config 2024-03-29T11:20:48.564 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:48.570+0000 7fc5b4a86640 10 monclient: build_initial_monmap 2024-03-29T11:20:48.564 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:48.570+0000 7fc5b4a86640 10 monclient: monmap: 2024-03-29T11:20:48.564 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-29T11:20:48.564 INFO:teuthology.misc.health.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:20:48.564 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T11:20:48.572459+0000 2024-03-29T11:20:48.564 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T11:20:48.572459+0000 2024-03-29T11:20:48.564 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-29T11:20:48.564 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:20:48.565 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-29T11:20:48.565 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.noname-c 2024-03-29T11:20:48.565 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-b 2024-03-29T11:20:48.565 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:20:48.565 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:48.571+0000 7fc5b4a86640 1 Processor -- start 2024-03-29T11:20:48.565 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:48.571+0000 7fc5b4a86640 1 -- start start 2024-03-29T11:20:48.565 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:48.571+0000 7fc5b4a86640 10 monclient: init 2024-03-29T11:20:48.565 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:48.572+0000 7fc5b4a86640 10 monclient: _reopen_session rank -1 2024-03-29T11:20:48.565 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:48.572+0000 7fc5b4a86640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:20:48.565 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:48.573+0000 7fc5b4a86640 10 monclient(hunting): picked mon.noname-a con 0x7fc5b0070170 addr v1:172.21.15.84:6789/0 2024-03-29T11:20:48.565 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:48.573+0000 7fc5ae59c640 1 --1- >> v1:172.21.15.84:6789/0 conn(0x7fc5b0070170 0x7fc5b013d4c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.84:6789/0 says I am v1:172.21.15.84:51526/0 (socket says 172.21.15.84:51526) 2024-03-29T11:20:48.565 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:48.573+0000 7fc5ae59c640 1 -- 172.21.15.84:0/2569485812 learned_addr learned my addr 172.21.15.84:0/2569485812 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-29T11:20:48.565 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:48.573+0000 7fc5b4a86640 10 monclient(hunting): picked mon.noname-b con 0x7fc5b00713f0 addr v1:172.21.15.84:6790/0 2024-03-29T11:20:48.565 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:48.573+0000 7fc5b4a86640 10 monclient(hunting): picked mon.noname-c con 0x7fc5b0070550 addr v1:172.21.15.179:6789/0 2024-03-29T11:20:48.566 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:48.573+0000 7fc5b4a86640 1 -- 172.21.15.84:0/2569485812 --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5b013dbd0 con 0x7fc5b0070170 2024-03-29T11:20:48.566 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:48.573+0000 7fc5b4a86640 1 -- 172.21.15.84:0/2569485812 --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5b013ff70 con 0x7fc5b0070550 2024-03-29T11:20:48.566 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:48.573+0000 7fc5b4a86640 1 -- 172.21.15.84:0/2569485812 --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5b0140250 con 0x7fc5b00713f0 2024-03-29T11:20:48.566 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:48.573+0000 7fc5b4a86640 10 monclient(hunting): _renew_subs 2024-03-29T11:20:48.566 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:48.573+0000 7fc5b4a86640 10 monclient(hunting): authenticate will time out at 909.868994s 2024-03-29T11:20:48.566 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:48.574+0000 7fc5ad59a640 1 -- 172.21.15.84:0/2569485812 <== mon.0 v1:172.21.15.84:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1257594502 0 0) 0x7fc5a4001110 con 0x7fc5b0070170 2024-03-29T11:20:48.566 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:48.574+0000 7fc5ad59a640 10 monclient(hunting): _negotiate 2024-03-29T11:20:48.566 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:48.574+0000 7fc5ad59a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:48.566 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:48.574+0000 7fc5ad59a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:48.566 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:48.574+0000 7fc5ad59a640 10 monclient(hunting): my global_id is 4281 2024-03-29T11:20:48.566 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:48.574+0000 7fc5ad59a640 1 -- 172.21.15.84:0/2569485812 --> v1:172.21.15.84:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc598003520 con 0x7fc5b0070170 2024-03-29T11:20:48.566 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:48.574+0000 7fc5ad59a640 1 -- 172.21.15.84:0/2569485812 <== mon.1 v1:172.21.15.179:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4194043550 0 0) 0x7fc5b013ff70 con 0x7fc5b0070550 2024-03-29T11:20:48.566 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:48.574+0000 7fc5ad59a640 10 monclient(hunting): _negotiate 2024-03-29T11:20:48.566 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:48.574+0000 7fc5ad59a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:48.567 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:48.574+0000 7fc5ad59a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:48.567 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:48.574+0000 7fc5ad59a640 10 monclient(hunting): my global_id is 4252 2024-03-29T11:20:48.567 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:48.574+0000 7fc5ad59a640 1 -- 172.21.15.84:0/2569485812 --> v1:172.21.15.179:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc5a4001110 con 0x7fc5b0070550 2024-03-29T11:20:48.567 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:48.574+0000 7fc5ad59a640 1 -- 172.21.15.84:0/2569485812 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1082202805 0 0) 0x7fc5b0140250 con 0x7fc5b00713f0 2024-03-29T11:20:48.567 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:48.574+0000 7fc5ad59a640 10 monclient(hunting): _negotiate 2024-03-29T11:20:48.567 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:48.574+0000 7fc5ad59a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:48.567 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:48.574+0000 7fc5ad59a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:48.567 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:48.574+0000 7fc5ad59a640 10 monclient(hunting): my global_id is 4280 2024-03-29T11:20:48.567 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:48.574+0000 7fc5ad59a640 1 -- 172.21.15.84:0/2569485812 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc5b013ff70 con 0x7fc5b00713f0 2024-03-29T11:20:48.567 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:48.575+0000 7fc5ad59a640 1 -- 172.21.15.84:0/2569485812 <== mon.1 v1:172.21.15.179:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 177662975 0 0) 0x7fc5a4001110 con 0x7fc5b0070550 2024-03-29T11:20:48.567 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:48.575+0000 7fc5ad59a640 1 -- 172.21.15.84:0/2569485812 >> v1:172.21.15.84:6790/0 conn(0x7fc5b00713f0 legacy=0x7fc5b01406e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:48.567 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:48.575+0000 7fc5ad59a640 1 -- 172.21.15.84:0/2569485812 >> v1:172.21.15.84:6789/0 conn(0x7fc5b0070170 legacy=0x7fc5b013d4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:48.567 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:48.575+0000 7fc5ad59a640 10 monclient: _finish_hunting 0 2024-03-29T11:20:48.567 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:48.575+0000 7fc5ad59a640 1 monclient: found mon.noname-c 2024-03-29T11:20:48.567 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:48.575+0000 7fc5ad59a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:20:48.568 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:48.575+0000 7fc5ad59a640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.179:6789/0 2024-03-29T11:20:48.568 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:48.575+0000 7fc5ad59a640 1 -- 172.21.15.84:0/2569485812 --> v1:172.21.15.179:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5b0140df0 con 0x7fc5b0070550 2024-03-29T11:20:48.568 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:48.575+0000 7fc5ad59a640 10 monclient: _finish_auth 0 2024-03-29T11:20:48.568 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:48.575+0000 7fc5ad59a640 10 monclient: _check_auth_tickets 2024-03-29T11:20:48.568 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:48.575+0000 7fc5ad59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:20:48.568 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:48.575+0000 7fc5ad59a640 1 -- 172.21.15.84:0/2569485812 <== mon.1 v1:172.21.15.179:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4236868694 0 0) 0x7fc5a0001670 con 0x7fc5b0070550 2024-03-29T11:20:48.568 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:48.575+0000 7fc5ad59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:48.568 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:48.575+0000 7fc5ad59a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:20:48.568 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:48.575+0000 7fc5ad59a640 10 monclient: dump: 2024-03-29T11:20:48.568 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T11:20:48.568 INFO:teuthology.misc.health.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:20:48.568 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T11:18:53.864025+0000 2024-03-29T11:20:48.568 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T11:18:53.864025+0000 2024-03-29T11:20:48.568 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:48.569 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:20:48.569 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T11:20:48.569 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T11:20:48.569 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T11:20:48.569 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:20:48.569 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:48.575+0000 7fc5b4a86640 5 monclient: authenticate success, global_id 4252 2024-03-29T11:20:48.569 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:48.575+0000 7fc5b4a86640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:20:48.569 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:48.575+0000 7fc5ad59a640 1 -- 172.21.15.84:0/2569485812 <== mon.1 v1:172.21.15.179:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc5a00017d0 con 0x7fc5b0070550 2024-03-29T11:20:48.569 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:48.575+0000 7fc5ad59a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:20:48.569 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:48.575+0000 7fc5ad59a640 1 -- 172.21.15.84:0/2569485812 <== mon.1 v1:172.21.15.179:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4236868694 0 0) 0x7fc5a0001be0 con 0x7fc5b0070550 2024-03-29T11:20:48.569 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:48.575+0000 7fc5ad59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:48.569 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:48.575+0000 7fc5ad59a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:20:48.569 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:48.575+0000 7fc5ad59a640 10 monclient: dump: 2024-03-29T11:20:48.569 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T11:20:48.570 INFO:teuthology.misc.health.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:20:48.570 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T11:18:53.864025+0000 2024-03-29T11:20:48.570 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T11:18:53.864025+0000 2024-03-29T11:20:48.570 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:48.570 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:20:48.570 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T11:20:48.570 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T11:20:48.570 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T11:20:48.570 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:20:48.570 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:48.575+0000 7fc5b4a86640 10 monclient: get_monmap_and_config success 2024-03-29T11:20:48.570 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:48.575+0000 7fc5b4a86640 4 set_mon_vals no callback set 2024-03-29T11:20:48.570 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:48.576+0000 7fc5b4a86640 10 monclient: shutdown 2024-03-29T11:20:48.570 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:48.576+0000 7fc5b4a86640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:20:48.570 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:48.576+0000 7fc5b4a86640 1 -- 172.21.15.84:0/2569485812 >> v1:172.21.15.179:6789/0 conn(0x7fc5b0070550 legacy=0x7fc5b014b7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:48.571 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:48.576+0000 7fc5b4a86640 1 -- 172.21.15.84:0/2569485812 shutdown_connections 2024-03-29T11:20:48.571 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:48.576+0000 7fc5b4a86640 1 -- 172.21.15.84:0/2569485812 >> 172.21.15.84:0/2569485812 conn(0x7fc5b0067610 msgr2=0x7fc5b00b5a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:20:48.571 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:48.576+0000 7fc5b4a86640 1 -- 172.21.15.84:0/2569485812 shutdown_connections 2024-03-29T11:20:48.571 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:48.576+0000 7fc5b4a86640 1 -- 172.21.15.84:0/2569485812 wait complete. 2024-03-29T11:20:48.571 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:48.578+0000 7fc5b4a86640 10 monclient: build_initial_monmap 2024-03-29T11:20:48.571 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:48.578+0000 7fc5b4a86640 10 monclient: monmap: 2024-03-29T11:20:48.571 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-29T11:20:48.571 INFO:teuthology.misc.health.smithi084.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:20:48.571 INFO:teuthology.misc.health.smithi084.stderr:last_changed 0.000000 2024-03-29T11:20:48.571 INFO:teuthology.misc.health.smithi084.stderr:created 0.000000 2024-03-29T11:20:48.571 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-29T11:20:48.571 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:20:48.571 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-29T11:20:48.571 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.noname-b 2024-03-29T11:20:48.571 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-c 2024-03-29T11:20:48.572 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:20:48.572 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:48.579+0000 7fc5b4a86640 1 librados: starting msgr at 2024-03-29T11:20:48.572 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:48.579+0000 7fc5b4a86640 1 librados: starting objecter 2024-03-29T11:20:48.572 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:48.579+0000 7fc5b4a86640 1 Processor -- start 2024-03-29T11:20:48.572 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:48.579+0000 7fc5b4a86640 1 -- start start 2024-03-29T11:20:48.572 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:48.579+0000 7fc5b4a86640 1 librados: setting wanted keys 2024-03-29T11:20:48.572 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:48.579+0000 7fc5b4a86640 1 librados: calling monclient init 2024-03-29T11:20:48.572 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:48.579+0000 7fc5b4a86640 10 monclient: init 2024-03-29T11:20:48.572 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:48.580+0000 7fc5b4a86640 10 monclient: _reopen_session rank -1 2024-03-29T11:20:48.572 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:48.580+0000 7fc5b4a86640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:20:48.572 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:48.580+0000 7fc5b4a86640 10 monclient(hunting): picked mon.noname-c con 0x7fc5b0070170 addr v1:172.21.15.84:6790/0 2024-03-29T11:20:48.572 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:48.580+0000 7fc5b4a86640 10 monclient(hunting): picked mon.noname-b con 0x7fc5b0070550 addr v1:172.21.15.179:6789/0 2024-03-29T11:20:48.572 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:48.580+0000 7fc5b4a86640 10 monclient(hunting): picked mon.noname-a con 0x7fc5b00713f0 addr v1:172.21.15.84:6789/0 2024-03-29T11:20:48.572 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:48.580+0000 7fc5b4a86640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5b012eee0 con 0x7fc5b00713f0 2024-03-29T11:20:48.572 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:48.580+0000 7fc5b4a86640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5b012f130 con 0x7fc5b0070550 2024-03-29T11:20:48.573 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:48.580+0000 7fc5b4a86640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5b012e6e0 con 0x7fc5b0070170 2024-03-29T11:20:48.573 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:48.580+0000 7fc5b4a86640 10 monclient(hunting): _renew_subs 2024-03-29T11:20:48.573 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:48.580+0000 7fc5b4a86640 10 monclient(hunting): authenticate will time out at 909.875752s 2024-03-29T11:20:48.573 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:48.580+0000 7fc5add9b640 1 --1- >> v1:172.21.15.179:6789/0 conn(0x7fc5b0070550 0x7fc5b012ab20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.179:6789/0 says I am v1:172.21.15.84:36420/0 (socket says 172.21.15.84:36420) 2024-03-29T11:20:48.573 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:48.580+0000 7fc5add9b640 1 -- 172.21.15.84:0/2782652098 learned_addr learned my addr 172.21.15.84:0/2782652098 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-29T11:20:48.573 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:48.581+0000 7fc59effd640 10 client.?.objecter ms_handle_connect 0x7fc5b0070550 2024-03-29T11:20:48.573 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:48.581+0000 7fc59effd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:20:48.573 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:48.581+0000 7fc59effd640 10 client.?.objecter ms_handle_connect 0x7fc5b00713f0 2024-03-29T11:20:48.573 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:48.581+0000 7fc59effd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:20:48.573 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:48.581+0000 7fc59effd640 1 -- 172.21.15.84:0/2782652098 <== mon.1 v1:172.21.15.179:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 323989949 0 0) 0x7fc5b012f130 con 0x7fc5b0070550 2024-03-29T11:20:48.573 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:48.581+0000 7fc59effd640 10 monclient(hunting): _negotiate 2024-03-29T11:20:48.573 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:48.581+0000 7fc59effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:48.573 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:48.581+0000 7fc59effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:48.573 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:48.581+0000 7fc59effd640 10 monclient(hunting): my global_id is 4255 2024-03-29T11:20:48.574 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:48.581+0000 7fc59effd640 1 -- 172.21.15.84:0/2782652098 --> v1:172.21.15.179:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc590004eb0 con 0x7fc5b0070550 2024-03-29T11:20:48.574 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:48.581+0000 7fc59effd640 10 client.?.objecter ms_handle_connect 0x7fc5b0070170 2024-03-29T11:20:48.574 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:48.581+0000 7fc59effd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:20:48.574 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:48.581+0000 7fc59effd640 1 -- 172.21.15.84:0/2782652098 <== mon.1 v1:172.21.15.179:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 378355954 0 0) 0x7fc590004eb0 con 0x7fc5b0070550 2024-03-29T11:20:48.574 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:48.581+0000 7fc59effd640 1 -- 172.21.15.84:0/2782652098 --> v1:172.21.15.179:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc5b012f130 con 0x7fc5b0070550 2024-03-29T11:20:48.574 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:48.582+0000 7fc59effd640 1 -- 172.21.15.84:0/2782652098 <== mon.1 v1:172.21.15.179:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4236868694 0 0) 0x7fc5a8001d50 con 0x7fc5b0070550 2024-03-29T11:20:48.574 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:48.582+0000 7fc59effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:48.574 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:48.582+0000 7fc59effd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:20:48.574 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:48.582+0000 7fc59effd640 10 monclient(hunting): dump: 2024-03-29T11:20:48.574 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T11:20:48.574 INFO:teuthology.misc.health.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:20:48.574 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T11:18:53.864025+0000 2024-03-29T11:20:48.574 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T11:18:53.864025+0000 2024-03-29T11:20:48.574 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:48.575 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:20:48.575 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T11:20:48.575 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T11:20:48.575 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T11:20:48.575 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:20:48.575 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:48.582+0000 7fc59effd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:20:48.575 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:48.582+0000 7fc59effd640 1 -- 172.21.15.84:0/2782652098 <== mon.0 v1:172.21.15.84:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3115793750 0 0) 0x7fc5b012eee0 con 0x7fc5b00713f0 2024-03-29T11:20:48.575 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:48.582+0000 7fc59effd640 10 monclient(hunting): _negotiate 2024-03-29T11:20:48.575 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:48.582+0000 7fc59effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:48.575 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:48.582+0000 7fc59effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:48.575 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:48.582+0000 7fc59effd640 10 monclient(hunting): my global_id is 4284 2024-03-29T11:20:48.575 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:48.582+0000 7fc59effd640 1 -- 172.21.15.84:0/2782652098 --> v1:172.21.15.84:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc590004eb0 con 0x7fc5b00713f0 2024-03-29T11:20:48.575 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:48.582+0000 7fc59effd640 1 -- 172.21.15.84:0/2782652098 <== mon.1 v1:172.21.15.179:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3311791655 0 0) 0x7fc5b012f130 con 0x7fc5b0070550 2024-03-29T11:20:48.575 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:48.582+0000 7fc59effd640 1 -- 172.21.15.84:0/2782652098 >> v1:172.21.15.84:6790/0 conn(0x7fc5b0070170 legacy=0x7fc5b01d5200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:48.575 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:48.582+0000 7fc59effd640 1 -- 172.21.15.84:0/2782652098 >> v1:172.21.15.84:6789/0 conn(0x7fc5b00713f0 legacy=0x7fc5b01e0a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:48.575 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:48.582+0000 7fc59effd640 10 monclient: _finish_hunting 0 2024-03-29T11:20:48.576 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:48.582+0000 7fc59effd640 1 monclient: found mon.b 2024-03-29T11:20:48.576 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:48.582+0000 7fc59effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:20:48.576 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:48.582+0000 7fc59effd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-29T11:20:48.576 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:48.582+0000 7fc59effd640 1 -- 172.21.15.84:0/2782652098 --> v1:172.21.15.179:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5b012fb40 con 0x7fc5b0070550 2024-03-29T11:20:48.576 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:48.582+0000 7fc59effd640 10 monclient: _finish_auth 0 2024-03-29T11:20:48.576 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:48.582+0000 7fc59effd640 10 monclient: _check_auth_tickets 2024-03-29T11:20:48.576 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:48.582+0000 7fc59effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:20:48.576 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:48.582+0000 7fc5b4a86640 5 monclient: authenticate success, global_id 4255 2024-03-29T11:20:48.576 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:48.582+0000 7fc5b4a86640 10 monclient: _renew_subs 2024-03-29T11:20:48.576 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:48.582+0000 7fc5b4a86640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-29T11:20:48.576 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:48.582+0000 7fc5b4a86640 1 -- 172.21.15.84:0/2782652098 --> v1:172.21.15.179:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc5b01d5910 con 0x7fc5b0070550 2024-03-29T11:20:48.576 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:48.582+0000 7fc5b4a86640 10 client.4255.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:20:48.576 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:48.582+0000 7fc5b4a86640 10 monclient: _renew_subs 2024-03-29T11:20:48.576 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:48.582+0000 7fc5b4a86640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-29T11:20:48.577 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:48.582+0000 7fc5b4a86640 1 -- 172.21.15.84:0/2782652098 --> v1:172.21.15.179:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc5b012f830 con 0x7fc5b0070550 2024-03-29T11:20:48.577 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:48.582+0000 7fc5b4a86640 1 librados: init done 2024-03-29T11:20:48.577 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:48.583+0000 7fc59effd640 1 -- 172.21.15.84:0/2782652098 <== mon.1 v1:172.21.15.179:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc5a8001f30 con 0x7fc5b0070550 2024-03-29T11:20:48.577 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:48.583+0000 7fc59effd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:20:48.577 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:48.583+0000 7fc5acd99640 4 set_mon_vals no callback set 2024-03-29T11:20:48.577 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:48.583+0000 7fc5b4a86640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:20:48.577 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:48.584+0000 7fc59effd640 1 -- 172.21.15.84:0/2782652098 <== mon.1 v1:172.21.15.179:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4236868694 0 0) 0x7fc5a8003de0 con 0x7fc5b0070550 2024-03-29T11:20:48.577 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:48.584+0000 7fc5b4a86640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:20:48.577 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:48.584+0000 7fc5b4a86640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-29T11:20:48.577 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:48.584+0000 7fc5b4a86640 1 -- 172.21.15.84:0/2782652098 --> v1:172.21.15.179:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc5b012f2c0 con 0x7fc5b0070550 2024-03-29T11:20:48.577 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:48.584+0000 7fc59effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:48.577 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:48.584+0000 7fc59effd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:20:48.577 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:48.584+0000 7fc59effd640 10 monclient: dump: 2024-03-29T11:20:48.577 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T11:20:48.577 INFO:teuthology.misc.health.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:20:48.577 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T11:18:53.864025+0000 2024-03-29T11:20:48.578 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T11:18:53.864025+0000 2024-03-29T11:20:48.578 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:48.578 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:20:48.578 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T11:20:48.578 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T11:20:48.578 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T11:20:48.578 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:20:48.578 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:48.585+0000 7fc59effd640 1 -- 172.21.15.84:0/2782652098 <== mon.1 v1:172.21.15.179:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4033942909 0 0) 0x7fc5a8003f40 con 0x7fc5b0070550 2024-03-29T11:20:48.578 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:48.586+0000 7fc59effd640 1 -- 172.21.15.84:0/2782652098 <== mon.1 v1:172.21.15.179:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2808139587 0 0) 0x7fc5a8084ef0 con 0x7fc5b0070550 2024-03-29T11:20:48.578 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:48.586+0000 7fc59effd640 10 client.4255.objecter ms_dispatch 0x7fc5b0000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:20:48.578 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:48.586+0000 7fc59effd640 3 client.4255.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:20:48.578 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:48.586+0000 7fc59effd640 3 client.4255.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:20:48.578 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:48.586+0000 7fc5ae59c640 10 monclient: get_auth_request con 0x7fc590072070 auth_method 0 2024-03-29T11:20:48.578 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:48.586+0000 7fc59effd640 20 client.4255.objecter dump_active .. 0 homeless 2024-03-29T11:20:48.579 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:48.586+0000 7fc59effd640 10 client.4255.objecter ms_handle_connect 0x7fc590072070 2024-03-29T11:20:48.579 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:48.591+0000 7fc59effd640 1 -- 172.21.15.84:0/2782652098 <== mon.1 v1:172.21.15.179:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fc5a80507d0 con 0x7fc5b0070550 2024-03-29T11:20:48.579 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:48.591+0000 7fc59effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:20:48.579 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:48.591+0000 7fc59effd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:20:48.953 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:48.975+0000 7fc5b4a86640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:20:48.953 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:48.975+0000 7fc5b4a86640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:20:48.953 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:48.976+0000 7fc5b4a86640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-29T11:20:48.954 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:48.976+0000 7fc5b4a86640 1 -- 172.21.15.84:0/2782652098 --> v1:172.21.15.179:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc5b0075f10 con 0x7fc5b0070550 2024-03-29T11:20:48.954 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:48.977+0000 7fc59effd640 1 -- 172.21.15.84:0/2782652098 <== mon.1 v1:172.21.15.179:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4209477549) 0x7fc5a8052700 con 0x7fc5b0070550 2024-03-29T11:20:48.954 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:48.977+0000 7fc59effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:20:48.954 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:48.977+0000 7fc59effd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:20:48.954 INFO:teuthology.misc.health.smithi084.stdout: 2024-03-29T11:20:48.954 INFO:teuthology.misc.health.smithi084.stdout:{"epoch":10,"fsid":"2c3e002f-9eb0-4860-853b-4ef1ef562213","created":"2024-03-29T11:19:32.029169+0000","modified":"2024-03-29T11:19:34.200299+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:34.200299+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"5c39e276-2034-4716-bbc0-57a634eefa92","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"cd580e99-109f-4bf1-88c0-a5987bc8b9ed","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"11e60a5a-051d-453f-a190-66387cb35592","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"2c329fbe-bbc4-4030-bf87-d711cad04679","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f4213d4a-bd36-4433-91ae-d7e2b02481d3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"f0cf44ea-6055-48af-805b-aed858102ff9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"d895eaa0-cc0b-4ee4-9b5a-2dfcd89a60b7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"9bc5a7bc-8f52-4db9-96d6-de5e68d52eaa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:20:48.958 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:48.980+0000 7fc5b4a86640 10 librados: watch_flush enter 2024-03-29T11:20:48.958 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:48.980+0000 7fc5b4a86640 10 librados: watch_flush exit 2024-03-29T11:20:48.958 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:48.980+0000 7fc5b4a86640 20 client.4255.objecter shutdown clearing up homeless session... 2024-03-29T11:20:48.958 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:48.980+0000 7fc5b4a86640 10 client.4255.objecter successfully canceled tick 2024-03-29T11:20:48.958 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:48.980+0000 7fc5b4a86640 1 -- 172.21.15.84:0/2782652098 >> v1:172.21.15.179:6816/2981134980 conn(0x7fc590072070 legacy=0x7fc590074530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:48.959 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:48.980+0000 7fc5b4a86640 10 monclient: shutdown 2024-03-29T11:20:48.959 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:48.980+0000 7fc5b4a86640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:20:48.959 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:48.980+0000 7fc5b4a86640 1 -- 172.21.15.84:0/2782652098 >> v1:172.21.15.179:6789/0 conn(0x7fc5b0070550 legacy=0x7fc5b012ab20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:48.959 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:48.980+0000 7fc5b4a86640 1 -- 172.21.15.84:0/2782652098 shutdown_connections 2024-03-29T11:20:48.959 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:48.980+0000 7fc5b4a86640 1 -- 172.21.15.84:0/2782652098 >> 172.21.15.84:0/2782652098 conn(0x7fc5b0067610 msgr2=0x7fc5b00b4e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:20:48.959 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:48.980+0000 7fc5b4a86640 1 -- 172.21.15.84:0/2782652098 shutdown_connections 2024-03-29T11:20:48.959 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:48.980+0000 7fc5b4a86640 1 -- 172.21.15.84:0/2782652098 wait complete. 2024-03-29T11:20:48.959 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:48.981+0000 7fc5b4a86640 1 librados: shutdown 2024-03-29T11:20:48.973 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:20:55.780 DEBUG:teuthology.orchestra.run.smithi084:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:20:55.911 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:55.934+0000 7f8f4fa25640 10 monclient: get_monmap_and_config 2024-03-29T11:20:55.911 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:55.934+0000 7f8f4fa25640 10 monclient: build_initial_monmap 2024-03-29T11:20:55.911 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:55.934+0000 7f8f4fa25640 10 monclient: monmap: 2024-03-29T11:20:55.911 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-29T11:20:55.911 INFO:teuthology.misc.health.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:20:55.911 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T11:20:55.935460+0000 2024-03-29T11:20:55.911 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T11:20:55.935460+0000 2024-03-29T11:20:55.911 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-29T11:20:55.911 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:20:55.911 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-29T11:20:55.912 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.noname-c 2024-03-29T11:20:55.912 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-b 2024-03-29T11:20:55.912 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:20:55.912 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:55.935+0000 7f8f4fa25640 1 Processor -- start 2024-03-29T11:20:55.912 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:55.936+0000 7f8f4fa25640 1 -- start start 2024-03-29T11:20:55.912 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:55.936+0000 7f8f4fa25640 10 monclient: init 2024-03-29T11:20:55.913 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:55.937+0000 7f8f4fa25640 10 monclient: _reopen_session rank -1 2024-03-29T11:20:55.913 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:55.937+0000 7f8f4fa25640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:20:55.913 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:55.937+0000 7f8f4fa25640 10 monclient(hunting): picked mon.noname-b con 0x7f8f481457f0 addr v1:172.21.15.84:6790/0 2024-03-29T11:20:55.913 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:55.937+0000 7f8f4fa25640 10 monclient(hunting): picked mon.noname-a con 0x7f8f4813f650 addr v1:172.21.15.84:6789/0 2024-03-29T11:20:55.914 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:55.937+0000 7f8f4fa25640 10 monclient(hunting): picked mon.noname-c con 0x7f8f480a9d70 addr v1:172.21.15.179:6789/0 2024-03-29T11:20:55.914 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:55.937+0000 7f8f4fa25640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8f480aaef0 con 0x7f8f4813f650 2024-03-29T11:20:55.914 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:55.937+0000 7f8f4fa25640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8f48145bd0 con 0x7f8f480a9d70 2024-03-29T11:20:55.914 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:55.937+0000 7f8f4fa25640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8f48145eb0 con 0x7f8f481457f0 2024-03-29T11:20:55.914 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:55.937+0000 7f8f4fa25640 10 monclient(hunting): _renew_subs 2024-03-29T11:20:55.914 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:55.937+0000 7f8f4fa25640 10 monclient(hunting): authenticate will time out at 917.231337s 2024-03-29T11:20:55.914 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:55.937+0000 7f8f4d7c1640 1 --1- >> v1:172.21.15.84:6790/0 conn(0x7f8f481457f0 0x7f8f4813ef40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.84:6790/0 says I am v1:172.21.15.84:50796/0 (socket says 172.21.15.84:50796) 2024-03-29T11:20:55.914 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:55.937+0000 7f8f4dfc2640 1 --1- >> v1:172.21.15.179:6789/0 conn(0x7f8f480a9d70 0x7f8f4814d8c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.179:6789/0 says I am v1:172.21.15.84:42868/0 (socket says 172.21.15.84:42868) 2024-03-29T11:20:55.914 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:55.937+0000 7f8f4d7c1640 1 -- 172.21.15.84:0/4064987536 learned_addr learned my addr 172.21.15.84:0/4064987536 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-29T11:20:55.914 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:55.937+0000 7f8f3ffff640 1 -- 172.21.15.84:0/4064987536 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 401325675 0 0) 0x7f8f48145eb0 con 0x7f8f481457f0 2024-03-29T11:20:55.914 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:55.937+0000 7f8f3ffff640 10 monclient(hunting): _negotiate 2024-03-29T11:20:55.914 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:55.937+0000 7f8f3ffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:55.914 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:55.937+0000 7f8f3ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:55.914 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:55.937+0000 7f8f3ffff640 10 monclient(hunting): my global_id is 4286 2024-03-29T11:20:55.915 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:55.937+0000 7f8f3ffff640 1 -- 172.21.15.84:0/4064987536 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8f34003520 con 0x7f8f481457f0 2024-03-29T11:20:55.915 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:55.937+0000 7f8f3ffff640 1 -- 172.21.15.84:0/4064987536 <== mon.2 v1:172.21.15.84:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2189119844 0 0) 0x7f8f34003520 con 0x7f8f481457f0 2024-03-29T11:20:55.915 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:55.937+0000 7f8f3ffff640 1 -- 172.21.15.84:0/4064987536 >> v1:172.21.15.179:6789/0 conn(0x7f8f480a9d70 legacy=0x7f8f4814d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:55.915 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:55.937+0000 7f8f3ffff640 1 -- 172.21.15.84:0/4064987536 >> v1:172.21.15.84:6789/0 conn(0x7f8f4813f650 legacy=0x7f8f480b2620 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:20:55.918 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:55.939+0000 7f8f3ffff640 10 monclient: _finish_hunting 0 2024-03-29T11:20:55.918 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:55.939+0000 7f8f3ffff640 1 monclient: found mon.noname-b 2024-03-29T11:20:55.918 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:55.939+0000 7f8f3ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:20:55.918 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:55.939+0000 7f8f3ffff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.84:6790/0 2024-03-29T11:20:55.918 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:55.939+0000 7f8f3ffff640 1 -- 172.21.15.84:0/4064987536 --> v1:172.21.15.84:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f48146340 con 0x7f8f481457f0 2024-03-29T11:20:55.919 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:55.939+0000 7f8f3ffff640 10 monclient: _finish_auth 0 2024-03-29T11:20:55.919 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:55.939+0000 7f8f3ffff640 10 monclient: _check_auth_tickets 2024-03-29T11:20:55.919 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:55.939+0000 7f8f3ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:20:55.919 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:55.939+0000 7f8f3ffff640 1 -- 172.21.15.84:0/4064987536 <== mon.2 v1:172.21.15.84:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4236868694 0 0) 0x7f8f40001690 con 0x7f8f481457f0 2024-03-29T11:20:55.919 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:55.939+0000 7f8f3ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:55.919 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:55.939+0000 7f8f3ffff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:20:55.919 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:55.939+0000 7f8f3ffff640 10 monclient: dump: 2024-03-29T11:20:55.919 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T11:20:55.919 INFO:teuthology.misc.health.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:20:55.919 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T11:18:53.864025+0000 2024-03-29T11:20:55.919 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T11:18:53.864025+0000 2024-03-29T11:20:55.919 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:55.919 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:20:55.919 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T11:20:55.919 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T11:20:55.920 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T11:20:55.920 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:20:55.920 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:55.940+0000 7f8f4fa25640 5 monclient: authenticate success, global_id 4286 2024-03-29T11:20:55.920 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:55.940+0000 7f8f4fa25640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:20:55.920 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:55.940+0000 7f8f3ffff640 1 -- 172.21.15.84:0/4064987536 <== mon.2 v1:172.21.15.84:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8f400018f0 con 0x7f8f481457f0 2024-03-29T11:20:55.920 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:55.940+0000 7f8f3ffff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:20:55.920 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:55.940+0000 7f8f3ffff640 1 -- 172.21.15.84:0/4064987536 <== mon.2 v1:172.21.15.84:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4236868694 0 0) 0x7f8f40001d00 con 0x7f8f481457f0 2024-03-29T11:20:55.920 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:55.940+0000 7f8f3ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:55.920 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:55.940+0000 7f8f3ffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:20:55.920 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:55.940+0000 7f8f3ffff640 10 monclient: dump: 2024-03-29T11:20:55.920 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T11:20:55.920 INFO:teuthology.misc.health.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:20:55.920 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T11:18:53.864025+0000 2024-03-29T11:20:55.920 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T11:18:53.864025+0000 2024-03-29T11:20:55.921 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:55.921 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:20:55.921 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T11:20:55.921 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T11:20:55.921 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T11:20:55.921 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:20:55.921 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:55.940+0000 7f8f4fa25640 10 monclient: get_monmap_and_config success 2024-03-29T11:20:55.921 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:55.940+0000 7f8f4fa25640 4 set_mon_vals no callback set 2024-03-29T11:20:55.921 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:55.941+0000 7f8f4fa25640 10 monclient: shutdown 2024-03-29T11:20:55.921 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:55.941+0000 7f8f4fa25640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:20:55.921 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:55.941+0000 7f8f4fa25640 1 -- 172.21.15.84:0/4064987536 >> v1:172.21.15.84:6790/0 conn(0x7f8f481457f0 legacy=0x7f8f4813ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:55.921 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:55.941+0000 7f8f4fa25640 1 -- 172.21.15.84:0/4064987536 shutdown_connections 2024-03-29T11:20:55.921 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:55.941+0000 7f8f4fa25640 1 -- 172.21.15.84:0/4064987536 >> 172.21.15.84:0/4064987536 conn(0x7f8f48066a00 msgr2=0x7f8f48066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:20:55.921 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:55.942+0000 7f8f4fa25640 1 -- 172.21.15.84:0/4064987536 shutdown_connections 2024-03-29T11:20:55.921 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:55.942+0000 7f8f4fa25640 1 -- 172.21.15.84:0/4064987536 wait complete. 2024-03-29T11:20:55.925 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:55.943+0000 7f8f4fa25640 10 monclient: build_initial_monmap 2024-03-29T11:20:55.925 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:55.943+0000 7f8f4fa25640 10 monclient: monmap: 2024-03-29T11:20:55.925 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-29T11:20:55.925 INFO:teuthology.misc.health.smithi084.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:20:55.925 INFO:teuthology.misc.health.smithi084.stderr:last_changed 0.000000 2024-03-29T11:20:55.925 INFO:teuthology.misc.health.smithi084.stderr:created 0.000000 2024-03-29T11:20:55.925 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-29T11:20:55.925 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:20:55.925 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-29T11:20:55.925 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.noname-b 2024-03-29T11:20:55.926 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-c 2024-03-29T11:20:55.926 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:20:55.926 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:55.943+0000 7f8f4fa25640 1 librados: starting msgr at 2024-03-29T11:20:55.926 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:55.943+0000 7f8f4fa25640 1 librados: starting objecter 2024-03-29T11:20:55.926 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:55.943+0000 7f8f4fa25640 1 Processor -- start 2024-03-29T11:20:55.926 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:55.943+0000 7f8f4fa25640 1 -- start start 2024-03-29T11:20:55.926 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:55.943+0000 7f8f4fa25640 1 librados: setting wanted keys 2024-03-29T11:20:55.926 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:55.943+0000 7f8f4fa25640 1 librados: calling monclient init 2024-03-29T11:20:55.926 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:55.943+0000 7f8f4fa25640 10 monclient: init 2024-03-29T11:20:55.926 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:55.944+0000 7f8f4fa25640 10 monclient: _reopen_session rank -1 2024-03-29T11:20:55.926 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:55.944+0000 7f8f4fa25640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:20:55.926 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:55.946+0000 7f8f4fa25640 10 monclient(hunting): picked mon.noname-a con 0x7f8f480a9d70 addr v1:172.21.15.84:6789/0 2024-03-29T11:20:55.926 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:55.946+0000 7f8f4fa25640 10 monclient(hunting): picked mon.noname-c con 0x7f8f4813f650 addr v1:172.21.15.84:6790/0 2024-03-29T11:20:55.926 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:55.946+0000 7f8f4fa25640 10 monclient(hunting): picked mon.noname-b con 0x7f8f481457f0 addr v1:172.21.15.179:6789/0 2024-03-29T11:20:55.927 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:55.946+0000 7f8f4fa25640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8f48141b60 con 0x7f8f480a9d70 2024-03-29T11:20:55.927 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:55.946+0000 7f8f4fa25640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8f48140480 con 0x7f8f481457f0 2024-03-29T11:20:55.927 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:55.946+0000 7f8f4fa25640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8f48140cd0 con 0x7f8f4813f650 2024-03-29T11:20:55.927 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:55.946+0000 7f8f4fa25640 10 monclient(hunting): _renew_subs 2024-03-29T11:20:55.927 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:55.946+0000 7f8f4fa25640 10 monclient(hunting): authenticate will time out at 917.240236s 2024-03-29T11:20:55.927 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:55.946+0000 7f8f4cfc0640 1 --1- >> v1:172.21.15.84:6790/0 conn(0x7f8f4813f650 0x7f8f481421f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.84:6790/0 says I am v1:172.21.15.84:51986/0 (socket says 172.21.15.84:51986) 2024-03-29T11:20:55.927 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:55.946+0000 7f8f4cfc0640 1 -- 172.21.15.84:0/3369051757 learned_addr learned my addr 172.21.15.84:0/3369051757 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-29T11:20:55.927 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:55.946+0000 7f8f3dffb640 10 client.?.objecter ms_handle_connect 0x7f8f481457f0 2024-03-29T11:20:55.927 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:55.946+0000 7f8f3dffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:20:55.927 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:55.946+0000 7f8f3dffb640 10 client.?.objecter ms_handle_connect 0x7f8f480a9d70 2024-03-29T11:20:55.927 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:55.946+0000 7f8f3dffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:20:55.927 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:55.946+0000 7f8f3dffb640 10 client.?.objecter ms_handle_connect 0x7f8f4813f650 2024-03-29T11:20:55.927 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:55.946+0000 7f8f3dffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:20:55.927 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:55.946+0000 7f8f3dffb640 1 -- 172.21.15.84:0/3369051757 <== mon.0 v1:172.21.15.84:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2381676699 0 0) 0x7f8f48141b60 con 0x7f8f480a9d70 2024-03-29T11:20:55.927 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:55.946+0000 7f8f3dffb640 10 monclient(hunting): _negotiate 2024-03-29T11:20:55.928 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:55.946+0000 7f8f3dffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:55.928 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:55.947+0000 7f8f3dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:55.928 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:55.947+0000 7f8f3dffb640 10 monclient(hunting): my global_id is 4287 2024-03-29T11:20:55.928 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:55.947+0000 7f8f3dffb640 1 -- 172.21.15.84:0/3369051757 --> v1:172.21.15.84:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8f2c004eb0 con 0x7f8f480a9d70 2024-03-29T11:20:55.928 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:55.947+0000 7f8f3dffb640 1 -- 172.21.15.84:0/3369051757 <== mon.1 v1:172.21.15.179:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3139739745 0 0) 0x7f8f48140480 con 0x7f8f481457f0 2024-03-29T11:20:55.928 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:55.947+0000 7f8f3dffb640 10 monclient(hunting): _negotiate 2024-03-29T11:20:55.928 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:55.947+0000 7f8f3dffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:55.928 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:55.947+0000 7f8f3dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:55.928 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:55.947+0000 7f8f3dffb640 10 monclient(hunting): my global_id is 4258 2024-03-29T11:20:55.928 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:55.947+0000 7f8f3dffb640 1 -- 172.21.15.84:0/3369051757 --> v1:172.21.15.179:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8f48141b60 con 0x7f8f481457f0 2024-03-29T11:20:55.928 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:55.947+0000 7f8f3dffb640 1 -- 172.21.15.84:0/3369051757 <== mon.0 v1:172.21.15.84:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3066786167 0 0) 0x7f8f2c004eb0 con 0x7f8f480a9d70 2024-03-29T11:20:55.928 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:55.947+0000 7f8f3dffb640 1 -- 172.21.15.84:0/3369051757 --> v1:172.21.15.84:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8f48140480 con 0x7f8f480a9d70 2024-03-29T11:20:55.928 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:55.947+0000 7f8f3dffb640 1 -- 172.21.15.84:0/3369051757 <== mon.0 v1:172.21.15.84:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4236868694 0 0) 0x7f8f40001650 con 0x7f8f480a9d70 2024-03-29T11:20:55.928 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:55.947+0000 7f8f3dffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:55.928 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:55.947+0000 7f8f3dffb640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:20:55.929 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:55.947+0000 7f8f3dffb640 10 monclient(hunting): dump: 2024-03-29T11:20:55.929 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T11:20:55.929 INFO:teuthology.misc.health.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:20:55.929 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T11:18:53.864025+0000 2024-03-29T11:20:55.929 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T11:18:53.864025+0000 2024-03-29T11:20:55.929 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:55.929 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:20:55.929 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T11:20:55.929 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T11:20:55.929 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T11:20:55.929 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:20:55.929 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:55.947+0000 7f8f3dffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:20:55.929 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:55.947+0000 7f8f3dffb640 1 -- 172.21.15.84:0/3369051757 <== mon.0 v1:172.21.15.84:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1649606368 0 0) 0x7f8f48140480 con 0x7f8f480a9d70 2024-03-29T11:20:55.929 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:55.947+0000 7f8f3dffb640 1 -- 172.21.15.84:0/3369051757 >> v1:172.21.15.84:6790/0 conn(0x7f8f4813f650 legacy=0x7f8f481421f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:55.929 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:55.947+0000 7f8f3dffb640 1 -- 172.21.15.84:0/3369051757 >> v1:172.21.15.179:6789/0 conn(0x7f8f481457f0 legacy=0x7f8f481e1d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:55.930 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:55.947+0000 7f8f3dffb640 10 monclient: _finish_hunting 0 2024-03-29T11:20:55.930 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:55.947+0000 7f8f3dffb640 1 monclient: found mon.a 2024-03-29T11:20:55.930 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:55.947+0000 7f8f3dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:20:55.930 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:55.947+0000 7f8f3dffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.84:6789/0 2024-03-29T11:20:55.930 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:55.947+0000 7f8f3dffb640 1 -- 172.21.15.84:0/3369051757 --> v1:172.21.15.84:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f4813dff0 con 0x7f8f480a9d70 2024-03-29T11:20:55.930 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:55.947+0000 7f8f3dffb640 10 monclient: _finish_auth 0 2024-03-29T11:20:55.930 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:55.947+0000 7f8f3dffb640 10 monclient: _check_auth_tickets 2024-03-29T11:20:55.930 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:55.947+0000 7f8f3dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:20:55.930 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:55.947+0000 7f8f4fa25640 5 monclient: authenticate success, global_id 4287 2024-03-29T11:20:55.930 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:55.947+0000 7f8f4fa25640 10 monclient: _renew_subs 2024-03-29T11:20:55.930 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:55.947+0000 7f8f4fa25640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.84:6789/0 2024-03-29T11:20:55.930 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:55.947+0000 7f8f4fa25640 1 -- 172.21.15.84:0/3369051757 --> v1:172.21.15.84:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8f4813e2a0 con 0x7f8f480a9d70 2024-03-29T11:20:55.930 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:55.948+0000 7f8f4fa25640 10 client.4287.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:20:55.931 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:55.948+0000 7f8f4fa25640 10 monclient: _renew_subs 2024-03-29T11:20:55.931 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:55.948+0000 7f8f4fa25640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.84:6789/0 2024-03-29T11:20:55.931 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:55.948+0000 7f8f4fa25640 1 -- 172.21.15.84:0/3369051757 --> v1:172.21.15.84:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f8f48142ae0 con 0x7f8f480a9d70 2024-03-29T11:20:55.931 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:55.948+0000 7f8f4fa25640 1 librados: init done 2024-03-29T11:20:55.933 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:55.948+0000 7f8f3dffb640 1 -- 172.21.15.84:0/3369051757 <== mon.0 v1:172.21.15.84:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8f40004a90 con 0x7f8f480a9d70 2024-03-29T11:20:55.933 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:55.948+0000 7f8f3dffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:20:55.933 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:55.948+0000 7f8f3dffb640 1 -- 172.21.15.84:0/3369051757 <== mon.0 v1:172.21.15.84:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4236868694 0 0) 0x7f8f40004da0 con 0x7f8f480a9d70 2024-03-29T11:20:55.933 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:55.948+0000 7f8f3dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:55.933 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:55.949+0000 7f8f3dffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:20:55.933 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:55.949+0000 7f8f3dffb640 10 monclient: dump: 2024-03-29T11:20:55.933 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T11:20:55.933 INFO:teuthology.misc.health.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:20:55.933 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T11:18:53.864025+0000 2024-03-29T11:20:55.933 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T11:18:53.864025+0000 2024-03-29T11:20:55.933 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:55.933 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:20:55.933 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T11:20:55.933 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T11:20:55.934 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T11:20:55.934 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:20:55.934 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:55.949+0000 7f8f3f7fe640 4 set_mon_vals no callback set 2024-03-29T11:20:55.934 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:55.950+0000 7f8f3dffb640 1 -- 172.21.15.84:0/3369051757 <== mon.0 v1:172.21.15.84:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4033942909 0 0) 0x7f8f4001ad70 con 0x7f8f480a9d70 2024-03-29T11:20:55.934 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:55.950+0000 7f8f4fa25640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:20:55.934 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:55.950+0000 7f8f4fa25640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:20:55.934 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:55.950+0000 7f8f4fa25640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.84:6789/0 2024-03-29T11:20:55.934 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:55.950+0000 7f8f4fa25640 1 -- 172.21.15.84:0/3369051757 --> v1:172.21.15.84:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8f28005080 con 0x7f8f480a9d70 2024-03-29T11:20:55.934 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:55.950+0000 7f8f3dffb640 1 -- 172.21.15.84:0/3369051757 <== mon.0 v1:172.21.15.84:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2808139587 0 0) 0x7f8f40084bb0 con 0x7f8f480a9d70 2024-03-29T11:20:55.934 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:55.951+0000 7f8f3dffb640 10 client.4287.objecter ms_dispatch 0x7f8f48000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:20:55.934 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:55.951+0000 7f8f3dffb640 3 client.4287.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:20:55.934 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:55.951+0000 7f8f3dffb640 3 client.4287.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:20:55.934 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:55.951+0000 7f8f3dffb640 20 client.4287.objecter dump_active .. 0 homeless 2024-03-29T11:20:55.934 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:55.953+0000 7f8f4cfc0640 10 monclient: get_auth_request con 0x7f8f2c06dd60 auth_method 0 2024-03-29T11:20:55.934 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:55.953+0000 7f8f3dffb640 10 client.4287.objecter ms_handle_connect 0x7f8f2c06dd60 2024-03-29T11:20:55.935 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:55.959+0000 7f8f3dffb640 1 -- 172.21.15.84:0/3369051757 <== mon.0 v1:172.21.15.84:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f8f40050490 con 0x7f8f480a9d70 2024-03-29T11:20:55.935 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:55.959+0000 7f8f3dffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:20:55.936 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:55.959+0000 7f8f3dffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:20:56.248 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:56.272+0000 7f8f4fa25640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:20:56.248 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:56.272+0000 7f8f4fa25640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:20:56.248 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:56.272+0000 7f8f4fa25640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.84:6789/0 2024-03-29T11:20:56.249 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:56.272+0000 7f8f4fa25640 1 -- 172.21.15.84:0/3369051757 --> v1:172.21.15.84:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8f28005330 con 0x7f8f480a9d70 2024-03-29T11:20:56.249 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:56.272+0000 7f8f3dffb640 1 -- 172.21.15.84:0/3369051757 <== mon.0 v1:172.21.15.84:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4209477549) 0x7f8f400523c0 con 0x7f8f480a9d70 2024-03-29T11:20:56.249 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:56.272+0000 7f8f3dffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:20:56.249 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:56.272+0000 7f8f3dffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:20:56.249 INFO:teuthology.misc.health.smithi084.stdout: 2024-03-29T11:20:56.250 INFO:teuthology.misc.health.smithi084.stdout:{"epoch":10,"fsid":"2c3e002f-9eb0-4860-853b-4ef1ef562213","created":"2024-03-29T11:19:32.029169+0000","modified":"2024-03-29T11:19:34.200299+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:34.200299+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"5c39e276-2034-4716-bbc0-57a634eefa92","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"cd580e99-109f-4bf1-88c0-a5987bc8b9ed","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"11e60a5a-051d-453f-a190-66387cb35592","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"2c329fbe-bbc4-4030-bf87-d711cad04679","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f4213d4a-bd36-4433-91ae-d7e2b02481d3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"f0cf44ea-6055-48af-805b-aed858102ff9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"d895eaa0-cc0b-4ee4-9b5a-2dfcd89a60b7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"9bc5a7bc-8f52-4db9-96d6-de5e68d52eaa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:20:56.252 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:56.275+0000 7f8f4fa25640 10 librados: watch_flush enter 2024-03-29T11:20:56.252 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:56.275+0000 7f8f4fa25640 10 librados: watch_flush exit 2024-03-29T11:20:56.252 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:56.275+0000 7f8f4fa25640 20 client.4287.objecter shutdown clearing up homeless session... 2024-03-29T11:20:56.253 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:56.275+0000 7f8f4fa25640 10 client.4287.objecter successfully canceled tick 2024-03-29T11:20:56.253 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:56.275+0000 7f8f4fa25640 1 -- 172.21.15.84:0/3369051757 >> v1:172.21.15.179:6816/2981134980 conn(0x7f8f2c06dd60 legacy=0x7f8f2c070220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:56.253 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:56.276+0000 7f8f4fa25640 10 monclient: shutdown 2024-03-29T11:20:56.253 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:56.276+0000 7f8f4fa25640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:20:56.253 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:56.276+0000 7f8f4fa25640 1 -- 172.21.15.84:0/3369051757 >> v1:172.21.15.84:6789/0 conn(0x7f8f480a9d70 legacy=0x7f8f48144dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:56.253 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:56.276+0000 7f8f4fa25640 1 -- 172.21.15.84:0/3369051757 shutdown_connections 2024-03-29T11:20:56.253 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:56.276+0000 7f8f4fa25640 1 -- 172.21.15.84:0/3369051757 >> 172.21.15.84:0/3369051757 conn(0x7f8f48066a00 msgr2=0x7f8f480b4820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:20:56.253 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:56.276+0000 7f8f4fa25640 1 -- 172.21.15.84:0/3369051757 shutdown_connections 2024-03-29T11:20:56.253 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:56.276+0000 7f8f4fa25640 1 -- 172.21.15.84:0/3369051757 wait complete. 2024-03-29T11:20:56.253 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:20:56.276+0000 7f8f4fa25640 1 librados: shutdown 2024-03-29T11:20:56.266 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:21:03.072 DEBUG:teuthology.orchestra.run.smithi084:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:21:03.223 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:03.247+0000 7f80c75d0640 10 monclient: get_monmap_and_config 2024-03-29T11:21:03.223 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:03.247+0000 7f80c75d0640 10 monclient: build_initial_monmap 2024-03-29T11:21:03.223 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:03.247+0000 7f80c75d0640 10 monclient: monmap: 2024-03-29T11:21:03.223 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-29T11:21:03.224 INFO:teuthology.misc.health.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:21:03.224 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T11:21:03.248099+0000 2024-03-29T11:21:03.224 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T11:21:03.248099+0000 2024-03-29T11:21:03.224 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-29T11:21:03.224 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:21:03.224 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-29T11:21:03.224 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.noname-c 2024-03-29T11:21:03.224 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-b 2024-03-29T11:21:03.224 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:21:03.225 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:03.248+0000 7f80c75d0640 1 Processor -- start 2024-03-29T11:21:03.225 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:03.248+0000 7f80c75d0640 1 -- start start 2024-03-29T11:21:03.225 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:03.248+0000 7f80c75d0640 10 monclient: init 2024-03-29T11:21:03.228 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:03.251+0000 7f80c75d0640 10 monclient: _reopen_session rank -1 2024-03-29T11:21:03.228 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:03.251+0000 7f80c75d0640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:21:03.228 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:03.251+0000 7f80c75d0640 10 monclient(hunting): picked mon.noname-c con 0x7f80b8006810 addr v1:172.21.15.179:6789/0 2024-03-29T11:21:03.228 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:03.251+0000 7f80c75d0640 10 monclient(hunting): picked mon.noname-b con 0x7f80b8006bf0 addr v1:172.21.15.84:6790/0 2024-03-29T11:21:03.228 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:03.251+0000 7f80c75d0640 10 monclient(hunting): picked mon.noname-a con 0x7f80b80c78f0 addr v1:172.21.15.84:6789/0 2024-03-29T11:21:03.228 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:03.251+0000 7f80c75d0640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f80b80c7f30 con 0x7f80b80c78f0 2024-03-29T11:21:03.229 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:03.251+0000 7f80c75d0640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f80b8036fd0 con 0x7f80b8006810 2024-03-29T11:21:03.229 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:03.251+0000 7f80c75d0640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f80b80c54f0 con 0x7f80b8006bf0 2024-03-29T11:21:03.229 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:03.251+0000 7f80c75d0640 10 monclient(hunting): _renew_subs 2024-03-29T11:21:03.229 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:03.251+0000 7f80c75d0640 10 monclient(hunting): authenticate will time out at 924.545775s 2024-03-29T11:21:03.229 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:03.251+0000 7f80c4b6b640 1 --1- >> v1:172.21.15.84:6790/0 conn(0x7f80b8006bf0 0x7f80b80cae50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.84:6790/0 says I am v1:172.21.15.84:54640/0 (socket says 172.21.15.84:54640) 2024-03-29T11:21:03.229 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:03.251+0000 7f80c4b6b640 1 -- 172.21.15.84:0/3244420075 learned_addr learned my addr 172.21.15.84:0/3244420075 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-29T11:21:03.229 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:03.253+0000 7f80b7fff640 1 -- 172.21.15.84:0/3244420075 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3663699082 0 0) 0x7f80b80c54f0 con 0x7f80b8006bf0 2024-03-29T11:21:03.229 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:03.253+0000 7f80b7fff640 10 monclient(hunting): _negotiate 2024-03-29T11:21:03.230 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:03.253+0000 7f80b7fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:21:03.230 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:03.253+0000 7f80b7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:21:03.230 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:03.253+0000 7f80b7fff640 10 monclient(hunting): my global_id is 4292 2024-03-29T11:21:03.230 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:03.253+0000 7f80b7fff640 1 -- 172.21.15.84:0/3244420075 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f80ac003520 con 0x7f80b8006bf0 2024-03-29T11:21:03.233 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:03.256+0000 7f80b7fff640 1 -- 172.21.15.84:0/3244420075 <== mon.0 v1:172.21.15.84:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4162318962 0 0) 0x7f80b80c7f30 con 0x7f80b80c78f0 2024-03-29T11:21:03.233 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:03.256+0000 7f80b7fff640 10 monclient(hunting): _negotiate 2024-03-29T11:21:03.233 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:03.256+0000 7f80b7fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:21:03.233 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:03.256+0000 7f80b7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:21:03.233 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:03.256+0000 7f80b7fff640 10 monclient(hunting): my global_id is 4290 2024-03-29T11:21:03.233 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:03.256+0000 7f80b7fff640 1 -- 172.21.15.84:0/3244420075 --> v1:172.21.15.84:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f80b80c54f0 con 0x7f80b80c78f0 2024-03-29T11:21:03.234 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:03.256+0000 7f80b7fff640 1 -- 172.21.15.84:0/3244420075 <== mon.1 v1:172.21.15.179:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2230217051 0 0) 0x7f80b8036fd0 con 0x7f80b8006810 2024-03-29T11:21:03.234 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:03.257+0000 7f80b7fff640 10 monclient(hunting): _negotiate 2024-03-29T11:21:03.234 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:03.257+0000 7f80b7fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:21:03.234 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:03.257+0000 7f80b7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:21:03.234 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:03.257+0000 7f80b7fff640 10 monclient(hunting): my global_id is 4261 2024-03-29T11:21:03.234 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:03.257+0000 7f80b7fff640 1 -- 172.21.15.84:0/3244420075 --> v1:172.21.15.179:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f80b80c7f30 con 0x7f80b8006810 2024-03-29T11:21:03.234 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:03.257+0000 7f80b7fff640 1 -- 172.21.15.84:0/3244420075 <== mon.2 v1:172.21.15.84:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1625402886 0 0) 0x7f80ac003520 con 0x7f80b8006bf0 2024-03-29T11:21:03.234 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:03.257+0000 7f80b7fff640 1 -- 172.21.15.84:0/3244420075 >> v1:172.21.15.179:6789/0 conn(0x7f80b8006810 legacy=0x7f80b8033620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:03.234 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:03.257+0000 7f80b7fff640 1 -- 172.21.15.84:0/3244420075 >> v1:172.21.15.84:6789/0 conn(0x7f80b80c78f0 legacy=0x7f80b80d31f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:03.234 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:03.257+0000 7f80b7fff640 10 monclient: _finish_hunting 0 2024-03-29T11:21:03.234 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:03.257+0000 7f80b7fff640 1 monclient: found mon.noname-b 2024-03-29T11:21:03.234 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:03.257+0000 7f80b7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:21:03.234 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:03.257+0000 7f80b7fff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.84:6790/0 2024-03-29T11:21:03.235 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:03.257+0000 7f80b7fff640 1 -- 172.21.15.84:0/3244420075 --> v1:172.21.15.84:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80b8037240 con 0x7f80b8006bf0 2024-03-29T11:21:03.235 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:03.257+0000 7f80b7fff640 10 monclient: _finish_auth 0 2024-03-29T11:21:03.235 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:03.257+0000 7f80b7fff640 10 monclient: _check_auth_tickets 2024-03-29T11:21:03.235 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:03.257+0000 7f80b7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:21:03.235 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:03.257+0000 7f80b7fff640 1 -- 172.21.15.84:0/3244420075 <== mon.2 v1:172.21.15.84:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4236868694 0 0) 0x7f80bc001690 con 0x7f80b8006bf0 2024-03-29T11:21:03.235 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:03.257+0000 7f80b7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:03.235 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:03.257+0000 7f80b7fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:21:03.235 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:03.257+0000 7f80b7fff640 10 monclient: dump: 2024-03-29T11:21:03.235 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T11:21:03.235 INFO:teuthology.misc.health.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:21:03.235 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T11:18:53.864025+0000 2024-03-29T11:21:03.235 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T11:18:53.864025+0000 2024-03-29T11:21:03.235 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:03.235 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:21:03.235 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T11:21:03.236 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T11:21:03.236 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T11:21:03.236 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:21:03.238 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:03.257+0000 7f80c75d0640 5 monclient: authenticate success, global_id 4292 2024-03-29T11:21:03.238 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:03.257+0000 7f80c75d0640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:21:03.238 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:03.257+0000 7f80b7fff640 1 -- 172.21.15.84:0/3244420075 <== mon.2 v1:172.21.15.84:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f80bc0018b0 con 0x7f80b8006bf0 2024-03-29T11:21:03.238 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:03.257+0000 7f80b7fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:21:03.239 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:03.257+0000 7f80b7fff640 1 -- 172.21.15.84:0/3244420075 <== mon.2 v1:172.21.15.84:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4236868694 0 0) 0x7f80bc001bc0 con 0x7f80b8006bf0 2024-03-29T11:21:03.239 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:03.257+0000 7f80b7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:03.239 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:03.257+0000 7f80b7fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:21:03.239 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:03.257+0000 7f80b7fff640 10 monclient: dump: 2024-03-29T11:21:03.239 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T11:21:03.239 INFO:teuthology.misc.health.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:21:03.239 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T11:18:53.864025+0000 2024-03-29T11:21:03.239 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T11:18:53.864025+0000 2024-03-29T11:21:03.239 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:03.239 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:21:03.239 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T11:21:03.239 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T11:21:03.239 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T11:21:03.239 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:21:03.240 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:03.257+0000 7f80c75d0640 10 monclient: get_monmap_and_config success 2024-03-29T11:21:03.240 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:03.257+0000 7f80c75d0640 4 set_mon_vals no callback set 2024-03-29T11:21:03.240 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:03.258+0000 7f80c75d0640 10 monclient: shutdown 2024-03-29T11:21:03.240 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:03.258+0000 7f80c75d0640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:21:03.240 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:03.258+0000 7f80c75d0640 1 -- 172.21.15.84:0/3244420075 >> v1:172.21.15.84:6790/0 conn(0x7f80b8006bf0 legacy=0x7f80b80cae50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:03.240 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:03.258+0000 7f80c75d0640 1 -- 172.21.15.84:0/3244420075 shutdown_connections 2024-03-29T11:21:03.240 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:03.258+0000 7f80c75d0640 1 -- 172.21.15.84:0/3244420075 >> 172.21.15.84:0/3244420075 conn(0x7f80b8005c40 msgr2=0x7f80b8006050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:21:03.240 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:03.258+0000 7f80c75d0640 1 -- 172.21.15.84:0/3244420075 shutdown_connections 2024-03-29T11:21:03.240 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:03.258+0000 7f80c75d0640 1 -- 172.21.15.84:0/3244420075 wait complete. 2024-03-29T11:21:03.240 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:03.260+0000 7f80c75d0640 10 monclient: build_initial_monmap 2024-03-29T11:21:03.240 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:03.260+0000 7f80c75d0640 10 monclient: monmap: 2024-03-29T11:21:03.240 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-29T11:21:03.240 INFO:teuthology.misc.health.smithi084.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:21:03.240 INFO:teuthology.misc.health.smithi084.stderr:last_changed 0.000000 2024-03-29T11:21:03.240 INFO:teuthology.misc.health.smithi084.stderr:created 0.000000 2024-03-29T11:21:03.241 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-29T11:21:03.241 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:21:03.241 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-29T11:21:03.241 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.noname-b 2024-03-29T11:21:03.241 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-c 2024-03-29T11:21:03.241 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:21:03.241 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:03.260+0000 7f80c75d0640 1 librados: starting msgr at 2024-03-29T11:21:03.241 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:03.262+0000 7f80c75d0640 1 librados: starting objecter 2024-03-29T11:21:03.241 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:03.262+0000 7f80c75d0640 1 Processor -- start 2024-03-29T11:21:03.241 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:03.262+0000 7f80c75d0640 1 -- start start 2024-03-29T11:21:03.241 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:03.262+0000 7f80c75d0640 1 librados: setting wanted keys 2024-03-29T11:21:03.241 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:03.262+0000 7f80c75d0640 1 librados: calling monclient init 2024-03-29T11:21:03.241 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:03.262+0000 7f80c75d0640 10 monclient: init 2024-03-29T11:21:03.242 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:03.263+0000 7f80c75d0640 10 monclient: _reopen_session rank -1 2024-03-29T11:21:03.242 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:03.264+0000 7f80c75d0640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:21:03.242 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:03.264+0000 7f80c75d0640 10 monclient(hunting): picked mon.noname-c con 0x7f80b8006810 addr v1:172.21.15.84:6790/0 2024-03-29T11:21:03.242 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:03.264+0000 7f80c75d0640 10 monclient(hunting): picked mon.noname-a con 0x7f80b80c78f0 addr v1:172.21.15.84:6789/0 2024-03-29T11:21:03.242 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:03.264+0000 7f80c75d0640 10 monclient(hunting): picked mon.noname-b con 0x7f80b815ffe0 addr v1:172.21.15.179:6789/0 2024-03-29T11:21:03.243 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:03.264+0000 7f80c75d0640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f80b802f040 con 0x7f80b80c78f0 2024-03-29T11:21:03.243 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:03.264+0000 7f80c75d0640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f80b815da10 con 0x7f80b815ffe0 2024-03-29T11:21:03.243 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:03.264+0000 7f80c75d0640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f80b8160500 con 0x7f80b8006810 2024-03-29T11:21:03.243 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:03.264+0000 7f80c75d0640 10 monclient(hunting): _renew_subs 2024-03-29T11:21:03.243 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:03.264+0000 7f80c75d0640 10 monclient(hunting): authenticate will time out at 924.558300s 2024-03-29T11:21:03.244 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:03.268+0000 7f80c536c640 1 --1- >> v1:172.21.15.84:6790/0 conn(0x7f80b8006810 0x7f80b815ca80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.84:6790/0 says I am v1:172.21.15.84:55248/0 (socket says 172.21.15.84:55248) 2024-03-29T11:21:03.244 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:03.268+0000 7f80c536c640 1 -- 172.21.15.84:0/2327845610 learned_addr learned my addr 172.21.15.84:0/2327845610 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-29T11:21:03.250 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:03.270+0000 7f80b5ffb640 10 client.?.objecter ms_handle_connect 0x7f80b8006810 2024-03-29T11:21:03.251 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:03.270+0000 7f80b5ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:21:03.251 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:03.270+0000 7f80b5ffb640 1 -- 172.21.15.84:0/2327845610 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3202888192 0 0) 0x7f80b8160500 con 0x7f80b8006810 2024-03-29T11:21:03.251 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:03.270+0000 7f80b5ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:21:03.251 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:03.270+0000 7f80b5ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:21:03.251 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:03.270+0000 7f80b5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:21:03.251 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:03.270+0000 7f80b5ffb640 10 monclient(hunting): my global_id is 4295 2024-03-29T11:21:03.251 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:03.270+0000 7f80b5ffb640 1 -- 172.21.15.84:0/2327845610 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f80a4004eb0 con 0x7f80b8006810 2024-03-29T11:21:03.251 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:03.271+0000 7f80b5ffb640 10 client.?.objecter ms_handle_connect 0x7f80b815ffe0 2024-03-29T11:21:03.251 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:03.271+0000 7f80b5ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:21:03.251 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:03.271+0000 7f80b5ffb640 10 client.?.objecter ms_handle_connect 0x7f80b80c78f0 2024-03-29T11:21:03.251 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:03.271+0000 7f80b5ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:21:03.251 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:03.271+0000 7f80b5ffb640 1 -- 172.21.15.84:0/2327845610 <== mon.1 v1:172.21.15.179:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2775409983 0 0) 0x7f80b80c54f0 con 0x7f80b815ffe0 2024-03-29T11:21:03.251 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:03.271+0000 7f80b5ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:21:03.251 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:03.271+0000 7f80b5ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:21:03.252 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:03.271+0000 7f80b5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:21:03.252 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:03.271+0000 7f80b5ffb640 10 monclient(hunting): my global_id is 4264 2024-03-29T11:21:03.252 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:03.271+0000 7f80b5ffb640 1 -- 172.21.15.84:0/2327845610 --> v1:172.21.15.179:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f80b8160500 con 0x7f80b815ffe0 2024-03-29T11:21:03.252 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:03.272+0000 7f80b5ffb640 1 -- 172.21.15.84:0/2327845610 <== mon.2 v1:172.21.15.84:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3345317930 0 0) 0x7f80a4004eb0 con 0x7f80b8006810 2024-03-29T11:21:03.252 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:03.272+0000 7f80b5ffb640 1 -- 172.21.15.84:0/2327845610 --> v1:172.21.15.84:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f80b80c54f0 con 0x7f80b8006810 2024-03-29T11:21:03.252 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:03.272+0000 7f80b5ffb640 1 -- 172.21.15.84:0/2327845610 <== mon.2 v1:172.21.15.84:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4236868694 0 0) 0x7f80b0001a10 con 0x7f80b8006810 2024-03-29T11:21:03.252 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:03.272+0000 7f80b5ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:03.252 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:03.272+0000 7f80b5ffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:21:03.252 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:03.272+0000 7f80b5ffb640 10 monclient(hunting): dump: 2024-03-29T11:21:03.252 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T11:21:03.252 INFO:teuthology.misc.health.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:21:03.252 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T11:18:53.864025+0000 2024-03-29T11:21:03.252 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T11:18:53.864025+0000 2024-03-29T11:21:03.252 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:03.253 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:21:03.253 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T11:21:03.253 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T11:21:03.253 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T11:21:03.253 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:21:03.253 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:03.272+0000 7f80b5ffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:21:03.253 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:03.272+0000 7f80b5ffb640 1 -- 172.21.15.84:0/2327845610 <== mon.1 v1:172.21.15.179:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3909054246 0 0) 0x7f80b8160500 con 0x7f80b815ffe0 2024-03-29T11:21:03.253 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:03.272+0000 7f80b5ffb640 1 -- 172.21.15.84:0/2327845610 --> v1:172.21.15.179:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f80a4004eb0 con 0x7f80b815ffe0 2024-03-29T11:21:03.254 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:03.272+0000 7f80b5ffb640 1 -- 172.21.15.84:0/2327845610 <== mon.1 v1:172.21.15.179:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4236868694 0 0) 0x7f80c0068810 con 0x7f80b815ffe0 2024-03-29T11:21:03.254 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:03.272+0000 7f80b5ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:03.254 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:03.272+0000 7f80b5ffb640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:21:03.254 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:03.272+0000 7f80b5ffb640 10 monclient(hunting): dump: 2024-03-29T11:21:03.254 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T11:21:03.254 INFO:teuthology.misc.health.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:21:03.254 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T11:18:53.864025+0000 2024-03-29T11:21:03.254 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T11:18:53.864025+0000 2024-03-29T11:21:03.254 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:03.254 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:21:03.254 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T11:21:03.254 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T11:21:03.254 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T11:21:03.254 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:21:03.255 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:03.272+0000 7f80b5ffb640 1 -- 172.21.15.84:0/2327845610 <== mon.2 v1:172.21.15.84:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2330519574 0 0) 0x7f80b80c54f0 con 0x7f80b8006810 2024-03-29T11:21:03.255 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:03.272+0000 7f80b5ffb640 1 -- 172.21.15.84:0/2327845610 >> v1:172.21.15.179:6789/0 conn(0x7f80b815ffe0 legacy=0x7f80b8159d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:03.255 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:03.272+0000 7f80b5ffb640 1 -- 172.21.15.84:0/2327845610 >> v1:172.21.15.84:6789/0 conn(0x7f80b80c78f0 legacy=0x7f80b802fa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:03.255 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:03.272+0000 7f80b5ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:21:03.255 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:03.272+0000 7f80b5ffb640 1 monclient: found mon.c 2024-03-29T11:21:03.255 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:03.272+0000 7f80b5ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:21:03.255 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:03.272+0000 7f80b5ffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-29T11:21:03.255 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:03.272+0000 7f80b5ffb640 1 -- 172.21.15.84:0/2327845610 --> v1:172.21.15.84:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80b802f270 con 0x7f80b8006810 2024-03-29T11:21:03.255 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:03.272+0000 7f80b5ffb640 10 monclient: _finish_auth 0 2024-03-29T11:21:03.255 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:03.272+0000 7f80b5ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:21:03.255 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:03.272+0000 7f80b5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:21:03.255 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:03.273+0000 7f80c75d0640 5 monclient: authenticate success, global_id 4295 2024-03-29T11:21:03.255 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:03.273+0000 7f80b5ffb640 1 -- 172.21.15.84:0/2327845610 <== mon.2 v1:172.21.15.84:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f80b0001cd0 con 0x7f80b8006810 2024-03-29T11:21:03.255 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:03.273+0000 7f80c75d0640 10 monclient: _renew_subs 2024-03-29T11:21:03.255 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:03.273+0000 7f80c75d0640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-29T11:21:03.256 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:03.273+0000 7f80c75d0640 1 -- 172.21.15.84:0/2327845610 --> v1:172.21.15.84:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f80b8030110 con 0x7f80b8006810 2024-03-29T11:21:03.256 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:03.273+0000 7f80b5ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:21:03.256 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:03.273+0000 7f80b5ffb640 1 -- 172.21.15.84:0/2327845610 <== mon.2 v1:172.21.15.84:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4236868694 0 0) 0x7f80b0003a60 con 0x7f80b8006810 2024-03-29T11:21:03.256 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:03.273+0000 7f80b5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:03.256 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:03.273+0000 7f80b5ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:21:03.256 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:03.273+0000 7f80b5ffb640 10 monclient: dump: 2024-03-29T11:21:03.256 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T11:21:03.256 INFO:teuthology.misc.health.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:21:03.256 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T11:18:53.864025+0000 2024-03-29T11:21:03.256 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T11:18:53.864025+0000 2024-03-29T11:21:03.256 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:03.256 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:21:03.256 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T11:21:03.256 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T11:21:03.257 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T11:21:03.257 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:21:03.257 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:03.273+0000 7f80c75d0640 10 client.4295.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:21:03.257 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:03.273+0000 7f80c75d0640 10 monclient: _renew_subs 2024-03-29T11:21:03.257 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:03.273+0000 7f80c75d0640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-29T11:21:03.257 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:03.273+0000 7f80c75d0640 1 -- 172.21.15.84:0/2327845610 --> v1:172.21.15.84:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f80b815a630 con 0x7f80b8006810 2024-03-29T11:21:03.257 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:03.273+0000 7f80c75d0640 1 librados: init done 2024-03-29T11:21:03.257 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:03.273+0000 7f80b77fe640 4 set_mon_vals no callback set 2024-03-29T11:21:03.257 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:03.274+0000 7f80c75d0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:21:03.258 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:03.275+0000 7f80c75d0640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:21:03.259 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:03.275+0000 7f80c75d0640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-29T11:21:03.259 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:03.275+0000 7f80c75d0640 1 -- 172.21.15.84:0/2327845610 --> v1:172.21.15.84:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f80b8004cf0 con 0x7f80b8006810 2024-03-29T11:21:03.259 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:03.276+0000 7f80b5ffb640 1 -- 172.21.15.84:0/2327845610 <== mon.2 v1:172.21.15.84:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4033942909 0 0) 0x7f80b0003bc0 con 0x7f80b8006810 2024-03-29T11:21:03.259 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:03.277+0000 7f80b5ffb640 1 -- 172.21.15.84:0/2327845610 <== mon.2 v1:172.21.15.84:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2808139587 0 0) 0x7f80b0084b80 con 0x7f80b8006810 2024-03-29T11:21:03.259 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:03.277+0000 7f80b5ffb640 10 client.4295.objecter ms_dispatch 0x7f80c0000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:21:03.259 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:03.277+0000 7f80b5ffb640 3 client.4295.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:21:03.259 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:03.277+0000 7f80b5ffb640 3 client.4295.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:21:03.259 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:03.277+0000 7f80b5ffb640 20 client.4295.objecter dump_active .. 0 homeless 2024-03-29T11:21:03.259 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:03.278+0000 7f80c4b6b640 10 monclient: get_auth_request con 0x7f80a406dda0 auth_method 0 2024-03-29T11:21:03.259 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:03.278+0000 7f80b5ffb640 10 client.4295.objecter ms_handle_connect 0x7f80a406dda0 2024-03-29T11:21:03.261 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:03.284+0000 7f80b5ffb640 1 -- 172.21.15.84:0/2327845610 <== mon.2 v1:172.21.15.84:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f80b0050460 con 0x7f80b8006810 2024-03-29T11:21:03.261 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:03.285+0000 7f80b5ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:21:03.261 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:03.285+0000 7f80b5ffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:21:03.586 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:03.609+0000 7f80c75d0640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:21:03.586 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:03.609+0000 7f80c75d0640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:21:03.586 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:03.609+0000 7f80c75d0640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-29T11:21:03.586 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:03.609+0000 7f80c75d0640 1 -- 172.21.15.84:0/2327845610 --> v1:172.21.15.84:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f80b8030480 con 0x7f80b8006810 2024-03-29T11:21:03.587 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:03.611+0000 7f80b5ffb640 1 -- 172.21.15.84:0/2327845610 <== mon.2 v1:172.21.15.84:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4209477549) 0x7f80b0052390 con 0x7f80b8006810 2024-03-29T11:21:03.587 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:03.611+0000 7f80b5ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:21:03.587 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:03.611+0000 7f80b5ffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:21:03.590 INFO:teuthology.misc.health.smithi084.stdout: 2024-03-29T11:21:03.590 INFO:teuthology.misc.health.smithi084.stdout:{"epoch":10,"fsid":"2c3e002f-9eb0-4860-853b-4ef1ef562213","created":"2024-03-29T11:19:32.029169+0000","modified":"2024-03-29T11:19:34.200299+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:34.200299+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"5c39e276-2034-4716-bbc0-57a634eefa92","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"cd580e99-109f-4bf1-88c0-a5987bc8b9ed","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"11e60a5a-051d-453f-a190-66387cb35592","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"2c329fbe-bbc4-4030-bf87-d711cad04679","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f4213d4a-bd36-4433-91ae-d7e2b02481d3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"f0cf44ea-6055-48af-805b-aed858102ff9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"d895eaa0-cc0b-4ee4-9b5a-2dfcd89a60b7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"9bc5a7bc-8f52-4db9-96d6-de5e68d52eaa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:21:03.592 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:03.616+0000 7f80c75d0640 10 librados: watch_flush enter 2024-03-29T11:21:03.592 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:03.616+0000 7f80c75d0640 10 librados: watch_flush exit 2024-03-29T11:21:03.593 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:03.616+0000 7f80c75d0640 20 client.4295.objecter shutdown clearing up homeless session... 2024-03-29T11:21:03.593 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:03.616+0000 7f80c75d0640 10 client.4295.objecter successfully canceled tick 2024-03-29T11:21:03.594 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:03.618+0000 7f80c75d0640 1 -- 172.21.15.84:0/2327845610 >> v1:172.21.15.179:6816/2981134980 conn(0x7f80a406dda0 legacy=0x7f80a4070260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:03.594 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:03.618+0000 7f80c75d0640 10 monclient: shutdown 2024-03-29T11:21:03.594 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:03.618+0000 7f80c75d0640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:21:03.594 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:03.618+0000 7f80c75d0640 1 -- 172.21.15.84:0/2327845610 >> v1:172.21.15.84:6790/0 conn(0x7f80b8006810 legacy=0x7f80b815ca80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:03.594 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:03.618+0000 7f80c75d0640 1 -- 172.21.15.84:0/2327845610 shutdown_connections 2024-03-29T11:21:03.595 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:03.618+0000 7f80c75d0640 1 -- 172.21.15.84:0/2327845610 >> 172.21.15.84:0/2327845610 conn(0x7f80b8005c40 msgr2=0x7f80b80043f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:21:03.595 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:03.618+0000 7f80c75d0640 1 -- 172.21.15.84:0/2327845610 shutdown_connections 2024-03-29T11:21:03.595 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:03.619+0000 7f80c75d0640 1 -- 172.21.15.84:0/2327845610 wait complete. 2024-03-29T11:21:03.595 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:03.619+0000 7f80c75d0640 1 librados: shutdown 2024-03-29T11:21:03.611 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:21:10.418 DEBUG:teuthology.orchestra.run.smithi084:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:21:10.572 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:10.595+0000 7f7dcfa25640 10 monclient: get_monmap_and_config 2024-03-29T11:21:10.572 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:10.595+0000 7f7dcfa25640 10 monclient: build_initial_monmap 2024-03-29T11:21:10.572 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:10.595+0000 7f7dcfa25640 10 monclient: monmap: 2024-03-29T11:21:10.572 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-29T11:21:10.572 INFO:teuthology.misc.health.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:21:10.572 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T11:21:10.596602+0000 2024-03-29T11:21:10.572 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T11:21:10.596602+0000 2024-03-29T11:21:10.572 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-29T11:21:10.572 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:21:10.572 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-29T11:21:10.572 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.noname-c 2024-03-29T11:21:10.573 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-b 2024-03-29T11:21:10.573 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:21:10.575 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:10.597+0000 7f7dcfa25640 1 Processor -- start 2024-03-29T11:21:10.575 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:10.597+0000 7f7dcfa25640 1 -- start start 2024-03-29T11:21:10.575 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:10.597+0000 7f7dcfa25640 10 monclient: init 2024-03-29T11:21:10.576 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:10.598+0000 7f7dcfa25640 10 monclient: _reopen_session rank -1 2024-03-29T11:21:10.576 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:10.598+0000 7f7dcfa25640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:21:10.576 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:10.598+0000 7f7dcfa25640 10 monclient(hunting): picked mon.noname-b con 0x7f7dc81457f0 addr v1:172.21.15.84:6790/0 2024-03-29T11:21:10.576 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:10.598+0000 7f7dcfa25640 10 monclient(hunting): picked mon.noname-c con 0x7f7dc813f650 addr v1:172.21.15.179:6789/0 2024-03-29T11:21:10.576 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:10.598+0000 7f7dcfa25640 10 monclient(hunting): picked mon.noname-a con 0x7f7dc80a9d70 addr v1:172.21.15.84:6789/0 2024-03-29T11:21:10.576 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:10.598+0000 7f7dcfa25640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7dc80aaef0 con 0x7f7dc80a9d70 2024-03-29T11:21:10.576 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:10.598+0000 7f7dcfa25640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7dc8145bd0 con 0x7f7dc813f650 2024-03-29T11:21:10.576 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:10.598+0000 7f7dcfa25640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7dc8145eb0 con 0x7f7dc81457f0 2024-03-29T11:21:10.576 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:10.598+0000 7f7dcfa25640 10 monclient(hunting): _renew_subs 2024-03-29T11:21:10.576 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:10.598+0000 7f7dcfa25640 10 monclient(hunting): authenticate will time out at 931.892285s 2024-03-29T11:21:10.579 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:10.603+0000 7f7dcd7c1640 1 --1- >> v1:172.21.15.84:6790/0 conn(0x7f7dc81457f0 0x7f7dc813ef40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.84:6790/0 says I am v1:172.21.15.84:39959/0 (socket says 172.21.15.84:39959) 2024-03-29T11:21:10.580 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:10.603+0000 7f7dcd7c1640 1 -- 172.21.15.84:0/1021617539 learned_addr learned my addr 172.21.15.84:0/1021617539 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-29T11:21:10.581 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:10.604+0000 7f7dbffff640 1 -- 172.21.15.84:0/1021617539 <== mon.0 v1:172.21.15.84:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4002312833 0 0) 0x7f7dc80aaef0 con 0x7f7dc80a9d70 2024-03-29T11:21:10.581 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:10.604+0000 7f7dbffff640 10 monclient(hunting): _negotiate 2024-03-29T11:21:10.581 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:10.604+0000 7f7dbffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:21:10.581 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:10.604+0000 7f7dbffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:21:10.581 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:10.604+0000 7f7dbffff640 10 monclient(hunting): my global_id is 4296 2024-03-29T11:21:10.581 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:10.604+0000 7f7dbffff640 1 -- 172.21.15.84:0/1021617539 --> v1:172.21.15.84:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7db4003520 con 0x7f7dc80a9d70 2024-03-29T11:21:10.585 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:10.606+0000 7f7dbffff640 1 -- 172.21.15.84:0/1021617539 <== mon.1 v1:172.21.15.179:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3907228837 0 0) 0x7f7dc8145bd0 con 0x7f7dc813f650 2024-03-29T11:21:10.585 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:10.606+0000 7f7dbffff640 10 monclient(hunting): _negotiate 2024-03-29T11:21:10.585 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:10.606+0000 7f7dbffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:21:10.585 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:10.606+0000 7f7dbffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:21:10.585 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:10.607+0000 7f7dbffff640 10 monclient(hunting): my global_id is 4267 2024-03-29T11:21:10.585 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:10.607+0000 7f7dbffff640 1 -- 172.21.15.84:0/1021617539 --> v1:172.21.15.179:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7dc80aaef0 con 0x7f7dc813f650 2024-03-29T11:21:10.585 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:10.607+0000 7f7dbffff640 1 -- 172.21.15.84:0/1021617539 <== mon.0 v1:172.21.15.84:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2520259803 0 0) 0x7f7db4003520 con 0x7f7dc80a9d70 2024-03-29T11:21:10.585 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:10.607+0000 7f7dbffff640 1 -- 172.21.15.84:0/1021617539 >> v1:172.21.15.84:6790/0 conn(0x7f7dc81457f0 legacy=0x7f7dc813ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:10.585 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:10.607+0000 7f7dbffff640 1 -- 172.21.15.84:0/1021617539 >> v1:172.21.15.179:6789/0 conn(0x7f7dc813f650 legacy=0x7f7dc80b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:10.585 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:10.607+0000 7f7dbffff640 10 monclient: _finish_hunting 0 2024-03-29T11:21:10.585 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:10.607+0000 7f7dbffff640 1 monclient: found mon.noname-a 2024-03-29T11:21:10.585 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:10.607+0000 7f7dbffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:21:10.585 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:10.607+0000 7f7dbffff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.84:6789/0 2024-03-29T11:21:10.586 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:10.607+0000 7f7dbffff640 1 -- 172.21.15.84:0/1021617539 --> v1:172.21.15.84:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7dc8146340 con 0x7f7dc80a9d70 2024-03-29T11:21:10.586 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:10.607+0000 7f7dbffff640 10 monclient: _finish_auth 0 2024-03-29T11:21:10.586 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:10.607+0000 7f7dbffff640 10 monclient: _check_auth_tickets 2024-03-29T11:21:10.586 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:10.607+0000 7f7dbffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:21:10.586 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:10.607+0000 7f7dbffff640 1 -- 172.21.15.84:0/1021617539 <== mon.0 v1:172.21.15.84:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4236868694 0 0) 0x7f7db8001690 con 0x7f7dc80a9d70 2024-03-29T11:21:10.586 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:10.607+0000 7f7dbffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:10.586 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:10.608+0000 7f7dbffff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:21:10.586 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:10.608+0000 7f7dbffff640 10 monclient: dump: 2024-03-29T11:21:10.586 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T11:21:10.586 INFO:teuthology.misc.health.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:21:10.586 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T11:18:53.864025+0000 2024-03-29T11:21:10.586 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T11:18:53.864025+0000 2024-03-29T11:21:10.587 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:10.587 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:21:10.587 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T11:21:10.587 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T11:21:10.587 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T11:21:10.587 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:21:10.587 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:10.608+0000 7f7dcfa25640 5 monclient: authenticate success, global_id 4296 2024-03-29T11:21:10.587 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:10.608+0000 7f7dcfa25640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:21:10.587 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:10.609+0000 7f7dbffff640 1 -- 172.21.15.84:0/1021617539 <== mon.0 v1:172.21.15.84:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7db80017f0 con 0x7f7dc80a9d70 2024-03-29T11:21:10.588 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:10.609+0000 7f7dbffff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:21:10.588 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:10.609+0000 7f7dbffff640 1 -- 172.21.15.84:0/1021617539 <== mon.0 v1:172.21.15.84:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4236868694 0 0) 0x7f7db8001c00 con 0x7f7dc80a9d70 2024-03-29T11:21:10.588 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:10.609+0000 7f7dbffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:10.588 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:10.609+0000 7f7dbffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:21:10.588 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:10.609+0000 7f7dbffff640 10 monclient: dump: 2024-03-29T11:21:10.588 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T11:21:10.588 INFO:teuthology.misc.health.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:21:10.588 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T11:18:53.864025+0000 2024-03-29T11:21:10.588 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T11:18:53.864025+0000 2024-03-29T11:21:10.588 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:10.588 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:21:10.588 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T11:21:10.588 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T11:21:10.588 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T11:21:10.589 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:21:10.589 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:10.609+0000 7f7dcfa25640 10 monclient: get_monmap_and_config success 2024-03-29T11:21:10.589 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:10.609+0000 7f7dcfa25640 4 set_mon_vals no callback set 2024-03-29T11:21:10.589 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:10.610+0000 7f7dcfa25640 10 monclient: shutdown 2024-03-29T11:21:10.589 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:10.610+0000 7f7dcfa25640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:21:10.589 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:10.610+0000 7f7dcfa25640 1 -- 172.21.15.84:0/1021617539 >> v1:172.21.15.84:6789/0 conn(0x7f7dc80a9d70 legacy=0x7f7dc814d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:10.589 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:10.612+0000 7f7dcfa25640 1 -- 172.21.15.84:0/1021617539 shutdown_connections 2024-03-29T11:21:10.589 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:10.612+0000 7f7dcfa25640 1 -- 172.21.15.84:0/1021617539 >> 172.21.15.84:0/1021617539 conn(0x7f7dc8066a00 msgr2=0x7f7dc8066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:21:10.590 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:10.614+0000 7f7dcfa25640 1 -- 172.21.15.84:0/1021617539 shutdown_connections 2024-03-29T11:21:10.593 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:10.617+0000 7f7dcfa25640 1 -- 172.21.15.84:0/1021617539 wait complete. 2024-03-29T11:21:10.596 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:10.618+0000 7f7dcfa25640 10 monclient: build_initial_monmap 2024-03-29T11:21:10.596 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:10.618+0000 7f7dcfa25640 10 monclient: monmap: 2024-03-29T11:21:10.596 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-29T11:21:10.596 INFO:teuthology.misc.health.smithi084.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:21:10.596 INFO:teuthology.misc.health.smithi084.stderr:last_changed 0.000000 2024-03-29T11:21:10.596 INFO:teuthology.misc.health.smithi084.stderr:created 0.000000 2024-03-29T11:21:10.596 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-29T11:21:10.596 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:21:10.596 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-29T11:21:10.596 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.noname-b 2024-03-29T11:21:10.596 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-c 2024-03-29T11:21:10.596 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:21:10.597 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:10.619+0000 7f7dcfa25640 1 librados: starting msgr at 2024-03-29T11:21:10.597 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:10.619+0000 7f7dcfa25640 1 librados: starting objecter 2024-03-29T11:21:10.597 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:10.619+0000 7f7dcfa25640 1 Processor -- start 2024-03-29T11:21:10.597 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:10.619+0000 7f7dcfa25640 1 -- start start 2024-03-29T11:21:10.597 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:10.619+0000 7f7dcfa25640 1 librados: setting wanted keys 2024-03-29T11:21:10.597 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:10.619+0000 7f7dcfa25640 1 librados: calling monclient init 2024-03-29T11:21:10.597 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:10.619+0000 7f7dcfa25640 10 monclient: init 2024-03-29T11:21:10.598 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:10.620+0000 7f7dcfa25640 10 monclient: _reopen_session rank -1 2024-03-29T11:21:10.598 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:10.620+0000 7f7dcfa25640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:21:10.598 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:10.620+0000 7f7dcfa25640 10 monclient(hunting): picked mon.noname-c con 0x7f7dc80a9d70 addr v1:172.21.15.84:6790/0 2024-03-29T11:21:10.598 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:10.620+0000 7f7dcfa25640 10 monclient(hunting): picked mon.noname-b con 0x7f7dc813f650 addr v1:172.21.15.179:6789/0 2024-03-29T11:21:10.598 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:10.620+0000 7f7dcfa25640 10 monclient(hunting): picked mon.noname-a con 0x7f7dc81457f0 addr v1:172.21.15.84:6789/0 2024-03-29T11:21:10.601 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:10.620+0000 7f7dcfa25640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7dc806de90 con 0x7f7dc81457f0 2024-03-29T11:21:10.601 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:10.620+0000 7f7dcfa25640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7dc806e5a0 con 0x7f7dc813f650 2024-03-29T11:21:10.601 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:10.620+0000 7f7dcfa25640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7dc806e320 con 0x7f7dc80a9d70 2024-03-29T11:21:10.601 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:10.620+0000 7f7dcfa25640 10 monclient(hunting): _renew_subs 2024-03-29T11:21:10.601 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:10.620+0000 7f7dcfa25640 10 monclient(hunting): authenticate will time out at 931.914819s 2024-03-29T11:21:10.602 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:10.625+0000 7f7dcdfc2640 1 --1- >> v1:172.21.15.84:6789/0 conn(0x7f7dc81457f0 0x7f7dc81db890 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.84:6789/0 says I am v1:172.21.15.84:54079/0 (socket says 172.21.15.84:54079) 2024-03-29T11:21:10.602 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:10.625+0000 7f7dcdfc2640 1 -- 172.21.15.84:0/392044607 learned_addr learned my addr 172.21.15.84:0/392044607 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-29T11:21:10.602 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:10.625+0000 7f7dbdffb640 10 client.?.objecter ms_handle_connect 0x7f7dc81457f0 2024-03-29T11:21:10.602 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:10.626+0000 7f7dbdffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:21:10.603 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:10.626+0000 7f7dbdffb640 1 -- 172.21.15.84:0/392044607 <== mon.0 v1:172.21.15.84:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3525185305 0 0) 0x7f7dc806de90 con 0x7f7dc81457f0 2024-03-29T11:21:10.603 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:10.626+0000 7f7dbdffb640 10 monclient(hunting): _negotiate 2024-03-29T11:21:10.603 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:10.626+0000 7f7dbdffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:21:10.607 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:10.626+0000 7f7dbdffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:21:10.607 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:10.626+0000 7f7dbdffb640 10 monclient(hunting): my global_id is 4299 2024-03-29T11:21:10.607 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:10.626+0000 7f7dbdffb640 1 -- 172.21.15.84:0/392044607 --> v1:172.21.15.84:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7dac004eb0 con 0x7f7dc81457f0 2024-03-29T11:21:10.607 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:10.627+0000 7f7dbdffb640 1 -- 172.21.15.84:0/392044607 <== mon.0 v1:172.21.15.84:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 727101864 0 0) 0x7f7dac004eb0 con 0x7f7dc81457f0 2024-03-29T11:21:10.607 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:10.627+0000 7f7dbdffb640 1 -- 172.21.15.84:0/392044607 --> v1:172.21.15.84:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7dac005300 con 0x7f7dc81457f0 2024-03-29T11:21:10.607 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:10.627+0000 7f7dbdffb640 1 -- 172.21.15.84:0/392044607 <== mon.0 v1:172.21.15.84:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4236868694 0 0) 0x7f7db8001b00 con 0x7f7dc81457f0 2024-03-29T11:21:10.607 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:10.627+0000 7f7dbdffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:10.607 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:10.627+0000 7f7dbdffb640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:21:10.607 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:10.627+0000 7f7dbdffb640 10 monclient(hunting): dump: 2024-03-29T11:21:10.607 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T11:21:10.607 INFO:teuthology.misc.health.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:21:10.608 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T11:18:53.864025+0000 2024-03-29T11:21:10.608 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T11:18:53.864025+0000 2024-03-29T11:21:10.608 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:10.608 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:21:10.608 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T11:21:10.608 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T11:21:10.608 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T11:21:10.608 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:21:10.608 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:10.627+0000 7f7dbdffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:21:10.608 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:10.630+0000 7f7dbdffb640 1 -- 172.21.15.84:0/392044607 <== mon.0 v1:172.21.15.84:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2779494027 0 0) 0x7f7dac005300 con 0x7f7dc81457f0 2024-03-29T11:21:10.608 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:10.630+0000 7f7dbdffb640 1 -- 172.21.15.84:0/392044607 >> v1:172.21.15.84:6790/0 conn(0x7f7dc80a9d70 legacy=0x7f7dc813b090 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:21:10.610 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:10.634+0000 7f7dbdffb640 1 -- 172.21.15.84:0/392044607 >> v1:172.21.15.179:6789/0 conn(0x7f7dc813f650 legacy=0x7f7dc8137880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:10.610 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:10.634+0000 7f7dbdffb640 10 monclient: _finish_hunting 0 2024-03-29T11:21:10.610 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:10.634+0000 7f7dbdffb640 1 monclient: found mon.a 2024-03-29T11:21:10.610 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:10.634+0000 7f7dbdffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:21:10.610 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:10.634+0000 7f7dbdffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.84:6789/0 2024-03-29T11:21:10.610 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:10.634+0000 7f7dbdffb640 1 -- 172.21.15.84:0/392044607 --> v1:172.21.15.84:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7dc806f6f0 con 0x7f7dc81457f0 2024-03-29T11:21:10.611 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:10.634+0000 7f7dbdffb640 10 monclient: _finish_auth 0 2024-03-29T11:21:10.611 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:10.634+0000 7f7dbdffb640 10 monclient: _check_auth_tickets 2024-03-29T11:21:10.611 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:10.634+0000 7f7dbdffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:21:10.611 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:10.634+0000 7f7dbdffb640 10 client.?.objecter ms_handle_connect 0x7f7dc813f650 2024-03-29T11:21:10.611 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:10.634+0000 7f7dbdffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:21:10.612 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:10.634+0000 7f7dcfa25640 5 monclient: authenticate success, global_id 4299 2024-03-29T11:21:10.612 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:10.634+0000 7f7dcfa25640 10 monclient: _renew_subs 2024-03-29T11:21:10.612 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:10.634+0000 7f7dcfa25640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.84:6789/0 2024-03-29T11:21:10.612 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:10.634+0000 7f7dcfa25640 1 -- 172.21.15.84:0/392044607 --> v1:172.21.15.84:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7dc8138cc0 con 0x7f7dc81457f0 2024-03-29T11:21:10.612 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:10.634+0000 7f7dcfa25640 10 client.4299.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:21:10.612 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:10.634+0000 7f7dcfa25640 10 monclient: _renew_subs 2024-03-29T11:21:10.613 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:10.634+0000 7f7dcfa25640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.84:6789/0 2024-03-29T11:21:10.613 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:10.634+0000 7f7dcfa25640 1 -- 172.21.15.84:0/392044607 --> v1:172.21.15.84:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f7dc8138170 con 0x7f7dc81457f0 2024-03-29T11:21:10.613 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:10.634+0000 7f7dcfa25640 1 librados: init done 2024-03-29T11:21:10.613 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:10.636+0000 7f7dbdffb640 1 -- 172.21.15.84:0/392044607 <== mon.0 v1:172.21.15.84:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7db8001c60 con 0x7f7dc81457f0 2024-03-29T11:21:10.613 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:10.636+0000 7f7dbdffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:21:10.613 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:10.636+0000 7f7dbdffb640 1 -- 172.21.15.84:0/392044607 <== mon.0 v1:172.21.15.84:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4236868694 0 0) 0x7f7db8003b00 con 0x7f7dc81457f0 2024-03-29T11:21:10.613 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:10.636+0000 7f7dbdffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:10.613 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:10.636+0000 7f7dbdffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:21:10.613 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:10.636+0000 7f7dbdffb640 10 monclient: dump: 2024-03-29T11:21:10.613 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T11:21:10.613 INFO:teuthology.misc.health.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:21:10.613 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T11:18:53.864025+0000 2024-03-29T11:21:10.613 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T11:18:53.864025+0000 2024-03-29T11:21:10.613 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:10.613 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:21:10.614 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T11:21:10.614 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T11:21:10.614 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T11:21:10.614 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:21:10.614 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:10.636+0000 7f7dbf7fe640 4 set_mon_vals no callback set 2024-03-29T11:21:10.614 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:10.637+0000 7f7dbdffb640 1 -- 172.21.15.84:0/392044607 <== mon.0 v1:172.21.15.84:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4033942909 0 0) 0x7f7db8003c60 con 0x7f7dc81457f0 2024-03-29T11:21:10.614 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:10.638+0000 7f7dbdffb640 1 -- 172.21.15.84:0/392044607 <== mon.0 v1:172.21.15.84:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2808139587 0 0) 0x7f7db8054890 con 0x7f7dc81457f0 2024-03-29T11:21:10.618 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:10.638+0000 7f7dbdffb640 10 client.4299.objecter ms_dispatch 0x7f7dc8000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:21:10.618 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:10.638+0000 7f7dbdffb640 3 client.4299.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:21:10.618 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:10.638+0000 7f7dbdffb640 3 client.4299.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:21:10.618 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:10.638+0000 7f7dbdffb640 20 client.4299.objecter dump_active .. 0 homeless 2024-03-29T11:21:10.618 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:10.640+0000 7f7dcfa25640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:21:10.618 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:10.640+0000 7f7dcfa25640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:21:10.619 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:10.640+0000 7f7dcfa25640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.84:6789/0 2024-03-29T11:21:10.619 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:10.640+0000 7f7dcfa25640 1 -- 172.21.15.84:0/392044607 --> v1:172.21.15.84:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7da8005080 con 0x7f7dc81457f0 2024-03-29T11:21:10.619 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:10.640+0000 7f7dcd7c1640 10 monclient: get_auth_request con 0x7f7dac06ddf0 auth_method 0 2024-03-29T11:21:10.619 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:10.641+0000 7f7dbdffb640 10 client.4299.objecter ms_handle_connect 0x7f7dac06ddf0 2024-03-29T11:21:10.628 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:10.652+0000 7f7dbdffb640 1 -- 172.21.15.84:0/392044607 <== mon.0 v1:172.21.15.84:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f7db8050b30 con 0x7f7dc81457f0 2024-03-29T11:21:10.628 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:10.652+0000 7f7dbdffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:21:10.628 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:10.652+0000 7f7dbdffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:21:10.959 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:10.980+0000 7f7dcfa25640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:21:10.959 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:10.981+0000 7f7dcfa25640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:21:10.959 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:10.981+0000 7f7dcfa25640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.84:6789/0 2024-03-29T11:21:10.959 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:10.981+0000 7f7dcfa25640 1 -- 172.21.15.84:0/392044607 --> v1:172.21.15.84:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7da8005330 con 0x7f7dc81457f0 2024-03-29T11:21:10.960 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:10.983+0000 7f7dbdffb640 1 -- 172.21.15.84:0/392044607 <== mon.0 v1:172.21.15.84:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4209477549) 0x7f7db8050cd0 con 0x7f7dc81457f0 2024-03-29T11:21:10.960 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:10.983+0000 7f7dbdffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:21:10.960 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:10.983+0000 7f7dbdffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:21:10.961 INFO:teuthology.misc.health.smithi084.stdout: 2024-03-29T11:21:10.961 INFO:teuthology.misc.health.smithi084.stdout:{"epoch":10,"fsid":"2c3e002f-9eb0-4860-853b-4ef1ef562213","created":"2024-03-29T11:19:32.029169+0000","modified":"2024-03-29T11:19:34.200299+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:34.200299+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"5c39e276-2034-4716-bbc0-57a634eefa92","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"cd580e99-109f-4bf1-88c0-a5987bc8b9ed","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"11e60a5a-051d-453f-a190-66387cb35592","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"2c329fbe-bbc4-4030-bf87-d711cad04679","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f4213d4a-bd36-4433-91ae-d7e2b02481d3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"f0cf44ea-6055-48af-805b-aed858102ff9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"d895eaa0-cc0b-4ee4-9b5a-2dfcd89a60b7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"9bc5a7bc-8f52-4db9-96d6-de5e68d52eaa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:21:10.963 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:10.987+0000 7f7dcfa25640 10 librados: watch_flush enter 2024-03-29T11:21:10.964 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:10.987+0000 7f7dcfa25640 10 librados: watch_flush exit 2024-03-29T11:21:10.964 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:10.987+0000 7f7dcfa25640 20 client.4299.objecter shutdown clearing up homeless session... 2024-03-29T11:21:10.964 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:10.987+0000 7f7dcfa25640 10 client.4299.objecter successfully canceled tick 2024-03-29T11:21:10.967 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:10.990+0000 7f7dcfa25640 1 -- 172.21.15.84:0/392044607 >> v1:172.21.15.179:6816/2981134980 conn(0x7f7dac06ddf0 legacy=0x7f7dac0702b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:10.967 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:10.990+0000 7f7dcfa25640 10 monclient: shutdown 2024-03-29T11:21:10.967 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:10.990+0000 7f7dcfa25640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:21:10.967 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:10.990+0000 7f7dcfa25640 1 -- 172.21.15.84:0/392044607 >> v1:172.21.15.84:6789/0 conn(0x7f7dc81457f0 legacy=0x7f7dc81db890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:10.967 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:10.990+0000 7f7dcfa25640 1 -- 172.21.15.84:0/392044607 shutdown_connections 2024-03-29T11:21:10.967 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:10.990+0000 7f7dcfa25640 1 -- 172.21.15.84:0/392044607 >> 172.21.15.84:0/392044607 conn(0x7f7dc8066a00 msgr2=0x7f7dc80b2520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:21:10.967 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:10.990+0000 7f7dcfa25640 1 -- 172.21.15.84:0/392044607 shutdown_connections 2024-03-29T11:21:10.968 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:10.991+0000 7f7dcfa25640 1 -- 172.21.15.84:0/392044607 wait complete. 2024-03-29T11:21:10.968 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:10.991+0000 7f7dcfa25640 1 librados: shutdown 2024-03-29T11:21:10.986 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:21:17.793 DEBUG:teuthology.orchestra.run.smithi084:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:21:17.944 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:17.966+0000 7fcbf0a4e640 10 monclient: get_monmap_and_config 2024-03-29T11:21:17.944 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:17.966+0000 7fcbf0a4e640 10 monclient: build_initial_monmap 2024-03-29T11:21:17.944 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:17.966+0000 7fcbf0a4e640 10 monclient: monmap: 2024-03-29T11:21:17.944 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-29T11:21:17.944 INFO:teuthology.misc.health.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:21:17.945 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T11:21:17.967424+0000 2024-03-29T11:21:17.945 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T11:21:17.967424+0000 2024-03-29T11:21:17.945 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-29T11:21:17.945 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:21:17.945 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-29T11:21:17.945 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.noname-c 2024-03-29T11:21:17.945 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-b 2024-03-29T11:21:17.945 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:21:17.946 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:17.969+0000 7fcbf0a4e640 1 Processor -- start 2024-03-29T11:21:17.946 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:17.969+0000 7fcbf0a4e640 1 -- start start 2024-03-29T11:21:17.946 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:17.969+0000 7fcbf0a4e640 10 monclient: init 2024-03-29T11:21:17.946 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:17.970+0000 7fcbf0a4e640 10 monclient: _reopen_session rank -1 2024-03-29T11:21:17.946 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:17.970+0000 7fcbf0a4e640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:21:17.947 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:17.970+0000 7fcbf0a4e640 10 monclient(hunting): picked mon.noname-a con 0x7fcbec0a9930 addr v1:172.21.15.84:6789/0 2024-03-29T11:21:17.947 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:17.970+0000 7fcbf0a4e640 10 monclient(hunting): picked mon.noname-c con 0x7fcbec149dc0 addr v1:172.21.15.179:6789/0 2024-03-29T11:21:17.947 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:17.970+0000 7fcbf0a4e640 10 monclient(hunting): picked mon.noname-b con 0x7fcbec14ff40 addr v1:172.21.15.84:6790/0 2024-03-29T11:21:17.947 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:17.970+0000 7fcbf0a4e640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcbec0aae60 con 0x7fcbec0a9930 2024-03-29T11:21:17.947 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:17.970+0000 7fcbf0a4e640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcbec0b4310 con 0x7fcbec149dc0 2024-03-29T11:21:17.947 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:17.970+0000 7fcbf0a4e640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcbec0afdd0 con 0x7fcbec14ff40 2024-03-29T11:21:17.947 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:17.970+0000 7fcbf0a4e640 10 monclient(hunting): _renew_subs 2024-03-29T11:21:17.947 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:17.970+0000 7fcbf0a4e640 10 monclient(hunting): authenticate will time out at 939.264939s 2024-03-29T11:21:17.951 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:17.975+0000 7fcbead9d640 1 --1- >> v1:172.21.15.84:6790/0 conn(0x7fcbec14ff40 0x7fcbec152330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.84:6790/0 says I am v1:172.21.15.84:40863/0 (socket says 172.21.15.84:40863) 2024-03-29T11:21:17.951 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:17.975+0000 7fcbead9d640 1 -- 172.21.15.84:0/632665275 learned_addr learned my addr 172.21.15.84:0/632665275 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-29T11:21:17.955 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:17.975+0000 7fcbe959a640 1 -- 172.21.15.84:0/632665275 <== mon.1 v1:172.21.15.179:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3951864106 0 0) 0x7fcbec0b4310 con 0x7fcbec149dc0 2024-03-29T11:21:17.955 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:17.975+0000 7fcbe959a640 10 monclient(hunting): _negotiate 2024-03-29T11:21:17.955 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:17.975+0000 7fcbe959a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:21:17.955 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:17.975+0000 7fcbe959a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:21:17.955 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:17.975+0000 7fcbe959a640 10 monclient(hunting): my global_id is 4273 2024-03-29T11:21:17.955 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:17.975+0000 7fcbe959a640 1 -- 172.21.15.84:0/632665275 --> v1:172.21.15.179:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcbd4003520 con 0x7fcbec149dc0 2024-03-29T11:21:17.955 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:17.976+0000 7fcbe959a640 1 -- 172.21.15.84:0/632665275 <== mon.1 v1:172.21.15.179:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3167300074 0 0) 0x7fcbd4003520 con 0x7fcbec149dc0 2024-03-29T11:21:17.955 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:17.976+0000 7fcbe959a640 1 -- 172.21.15.84:0/632665275 >> v1:172.21.15.84:6790/0 conn(0x7fcbec14ff40 legacy=0x7fcbec152330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:17.955 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:17.976+0000 7fcbe959a640 1 -- 172.21.15.84:0/632665275 >> v1:172.21.15.84:6789/0 conn(0x7fcbec0a9930 legacy=0x7fcbec143a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:21:17.955 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:17.976+0000 7fcbe959a640 10 monclient: _finish_hunting 0 2024-03-29T11:21:17.955 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:17.976+0000 7fcbe959a640 1 monclient: found mon.noname-c 2024-03-29T11:21:17.956 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:17.976+0000 7fcbe959a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:21:17.956 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:17.976+0000 7fcbe959a640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.179:6789/0 2024-03-29T11:21:17.956 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:17.976+0000 7fcbe959a640 1 -- 172.21.15.84:0/632665275 --> v1:172.21.15.179:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcbec14a950 con 0x7fcbec149dc0 2024-03-29T11:21:17.956 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:17.976+0000 7fcbe959a640 10 monclient: _finish_auth 0 2024-03-29T11:21:17.956 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:17.976+0000 7fcbe959a640 10 monclient: _check_auth_tickets 2024-03-29T11:21:17.956 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:17.976+0000 7fcbe959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:21:17.956 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:17.976+0000 7fcbe959a640 1 -- 172.21.15.84:0/632665275 <== mon.1 v1:172.21.15.179:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4236868694 0 0) 0x7fcbe4001a10 con 0x7fcbec149dc0 2024-03-29T11:21:17.956 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:17.976+0000 7fcbe959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:17.956 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:17.976+0000 7fcbe959a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:21:17.956 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:17.976+0000 7fcbe959a640 10 monclient: dump: 2024-03-29T11:21:17.956 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T11:21:17.956 INFO:teuthology.misc.health.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:21:17.956 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T11:18:53.864025+0000 2024-03-29T11:21:17.956 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T11:18:53.864025+0000 2024-03-29T11:21:17.956 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:17.957 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:21:17.957 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T11:21:17.957 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T11:21:17.957 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T11:21:17.957 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:21:17.957 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:17.976+0000 7fcbf0a4e640 5 monclient: authenticate success, global_id 4273 2024-03-29T11:21:17.957 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:17.976+0000 7fcbf0a4e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:21:17.957 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:17.976+0000 7fcbe959a640 1 -- 172.21.15.84:0/632665275 <== mon.1 v1:172.21.15.179:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fcbe4001ce0 con 0x7fcbec149dc0 2024-03-29T11:21:17.957 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:17.976+0000 7fcbe959a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:21:17.957 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:17.976+0000 7fcbe959a640 1 -- 172.21.15.84:0/632665275 <== mon.1 v1:172.21.15.179:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4236868694 0 0) 0x7fcbe4002070 con 0x7fcbec149dc0 2024-03-29T11:21:17.957 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:17.976+0000 7fcbe959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:17.957 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:17.976+0000 7fcbe959a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:21:17.957 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:17.976+0000 7fcbe959a640 10 monclient: dump: 2024-03-29T11:21:17.957 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T11:21:17.957 INFO:teuthology.misc.health.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:21:17.958 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T11:18:53.864025+0000 2024-03-29T11:21:17.958 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T11:18:53.864025+0000 2024-03-29T11:21:17.958 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:17.958 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:21:17.958 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T11:21:17.958 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T11:21:17.958 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T11:21:17.958 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:21:17.958 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:17.976+0000 7fcbf0a4e640 10 monclient: get_monmap_and_config success 2024-03-29T11:21:17.958 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:17.976+0000 7fcbf0a4e640 4 set_mon_vals no callback set 2024-03-29T11:21:17.958 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:17.977+0000 7fcbf0a4e640 10 monclient: shutdown 2024-03-29T11:21:17.958 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:17.977+0000 7fcbf0a4e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:21:17.958 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:17.977+0000 7fcbf0a4e640 1 -- 172.21.15.84:0/632665275 >> v1:172.21.15.179:6789/0 conn(0x7fcbec149dc0 legacy=0x7fcbec14a240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:17.959 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:17.981+0000 7fcbf0a4e640 1 -- 172.21.15.84:0/632665275 shutdown_connections 2024-03-29T11:21:17.959 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:17.981+0000 7fcbf0a4e640 1 -- 172.21.15.84:0/632665275 >> 172.21.15.84:0/632665275 conn(0x7fcbec0b35e0 msgr2=0x7fcbec0b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:21:17.959 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:17.983+0000 7fcbf0a4e640 1 -- 172.21.15.84:0/632665275 shutdown_connections 2024-03-29T11:21:17.960 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:17.983+0000 7fcbf0a4e640 1 -- 172.21.15.84:0/632665275 wait complete. 2024-03-29T11:21:17.961 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:17.984+0000 7fcbf0a4e640 10 monclient: build_initial_monmap 2024-03-29T11:21:17.961 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:17.984+0000 7fcbf0a4e640 10 monclient: monmap: 2024-03-29T11:21:17.961 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-29T11:21:17.961 INFO:teuthology.misc.health.smithi084.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:21:17.961 INFO:teuthology.misc.health.smithi084.stderr:last_changed 0.000000 2024-03-29T11:21:17.961 INFO:teuthology.misc.health.smithi084.stderr:created 0.000000 2024-03-29T11:21:17.961 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-29T11:21:17.961 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:21:17.961 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-29T11:21:17.961 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.noname-b 2024-03-29T11:21:17.961 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-c 2024-03-29T11:21:17.961 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:21:17.962 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:17.985+0000 7fcbf0a4e640 1 librados: starting msgr at 2024-03-29T11:21:17.962 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:17.985+0000 7fcbf0a4e640 1 librados: starting objecter 2024-03-29T11:21:17.962 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:17.985+0000 7fcbf0a4e640 1 Processor -- start 2024-03-29T11:21:17.962 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:17.985+0000 7fcbf0a4e640 1 -- start start 2024-03-29T11:21:17.962 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:17.985+0000 7fcbf0a4e640 1 librados: setting wanted keys 2024-03-29T11:21:17.962 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:17.985+0000 7fcbf0a4e640 1 librados: calling monclient init 2024-03-29T11:21:17.962 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:17.985+0000 7fcbf0a4e640 10 monclient: init 2024-03-29T11:21:17.965 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:17.986+0000 7fcbf0a4e640 10 monclient: _reopen_session rank -1 2024-03-29T11:21:17.965 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:17.986+0000 7fcbf0a4e640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:21:17.965 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:17.986+0000 7fcbf0a4e640 10 monclient(hunting): picked mon.noname-c con 0x7fcbec0a9930 addr v1:172.21.15.84:6790/0 2024-03-29T11:21:17.965 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:17.986+0000 7fcbf0a4e640 10 monclient(hunting): picked mon.noname-b con 0x7fcbec149dc0 addr v1:172.21.15.179:6789/0 2024-03-29T11:21:17.965 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:17.986+0000 7fcbf0a4e640 10 monclient(hunting): picked mon.noname-a con 0x7fcbec14ff40 addr v1:172.21.15.84:6789/0 2024-03-29T11:21:17.965 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:17.986+0000 7fcbf0a4e640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcbec06fff0 con 0x7fcbec14ff40 2024-03-29T11:21:17.965 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:17.986+0000 7fcbf0a4e640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcbec149a00 con 0x7fcbec149dc0 2024-03-29T11:21:17.965 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:17.986+0000 7fcbf0a4e640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcbec1dbef0 con 0x7fcbec0a9930 2024-03-29T11:21:17.965 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:17.986+0000 7fcbf0a4e640 10 monclient(hunting): _renew_subs 2024-03-29T11:21:17.965 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:17.986+0000 7fcbf0a4e640 10 monclient(hunting): authenticate will time out at 939.280213s 2024-03-29T11:21:17.968 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:17.992+0000 7fcbe9d9b640 1 --1- >> v1:172.21.15.179:6789/0 conn(0x7fcbec149dc0 0x7fcbec1db7e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.179:6789/0 says I am v1:172.21.15.84:48199/0 (socket says 172.21.15.84:48199) 2024-03-29T11:21:17.968 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:17.992+0000 7fcbe9d9b640 1 -- 172.21.15.84:0/664901756 learned_addr learned my addr 172.21.15.84:0/664901756 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-29T11:21:17.969 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:17.992+0000 7fcbdaffd640 10 client.?.objecter ms_handle_connect 0x7fcbec14ff40 2024-03-29T11:21:17.969 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:17.992+0000 7fcbdaffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:21:17.971 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:17.994+0000 7fcbdaffd640 1 -- 172.21.15.84:0/664901756 <== mon.0 v1:172.21.15.84:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2323744977 0 0) 0x7fcbec06fff0 con 0x7fcbec14ff40 2024-03-29T11:21:17.971 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:17.994+0000 7fcbdaffd640 10 monclient(hunting): _negotiate 2024-03-29T11:21:17.972 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:17.994+0000 7fcbdaffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:21:17.972 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:17.994+0000 7fcbdaffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:21:17.972 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:17.994+0000 7fcbdaffd640 10 monclient(hunting): my global_id is 4302 2024-03-29T11:21:17.972 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:17.994+0000 7fcbdaffd640 1 -- 172.21.15.84:0/664901756 --> v1:172.21.15.84:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcbcc004eb0 con 0x7fcbec14ff40 2024-03-29T11:21:17.973 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:17.994+0000 7fcbdaffd640 1 -- 172.21.15.84:0/664901756 <== mon.0 v1:172.21.15.84:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4081647279 0 0) 0x7fcbcc004eb0 con 0x7fcbec14ff40 2024-03-29T11:21:17.973 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:17.994+0000 7fcbdaffd640 1 -- 172.21.15.84:0/664901756 --> v1:172.21.15.84:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fcbcc005300 con 0x7fcbec14ff40 2024-03-29T11:21:17.973 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:17.994+0000 7fcbdaffd640 1 -- 172.21.15.84:0/664901756 <== mon.0 v1:172.21.15.84:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4236868694 0 0) 0x7fcbe0001740 con 0x7fcbec14ff40 2024-03-29T11:21:17.973 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:17.994+0000 7fcbdaffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:17.973 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:17.994+0000 7fcbdaffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:21:17.973 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:17.994+0000 7fcbdaffd640 10 monclient(hunting): dump: 2024-03-29T11:21:17.973 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T11:21:17.973 INFO:teuthology.misc.health.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:21:17.973 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T11:18:53.864025+0000 2024-03-29T11:21:17.973 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T11:18:53.864025+0000 2024-03-29T11:21:17.973 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:17.973 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:21:17.973 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T11:21:17.973 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T11:21:17.974 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T11:21:17.974 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:21:17.974 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:17.994+0000 7fcbdaffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:21:17.974 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:17.995+0000 7fcbdaffd640 1 -- 172.21.15.84:0/664901756 <== mon.0 v1:172.21.15.84:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2284267571 0 0) 0x7fcbcc005300 con 0x7fcbec14ff40 2024-03-29T11:21:17.974 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:17.995+0000 7fcbdaffd640 1 -- 172.21.15.84:0/664901756 >> v1:172.21.15.84:6790/0 conn(0x7fcbec0a9930 legacy=0x7fcbec13f280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:17.974 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:17.995+0000 7fcbdaffd640 1 -- 172.21.15.84:0/664901756 >> v1:172.21.15.179:6789/0 conn(0x7fcbec149dc0 legacy=0x7fcbec1db7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:17.974 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:17.995+0000 7fcbdaffd640 10 monclient: _finish_hunting 0 2024-03-29T11:21:17.974 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:17.995+0000 7fcbdaffd640 1 monclient: found mon.a 2024-03-29T11:21:17.974 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:17.995+0000 7fcbdaffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:21:17.974 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:17.995+0000 7fcbdaffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.84:6789/0 2024-03-29T11:21:17.974 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:17.995+0000 7fcbdaffd640 1 -- 172.21.15.84:0/664901756 --> v1:172.21.15.84:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcbec06fb20 con 0x7fcbec14ff40 2024-03-29T11:21:17.974 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:17.995+0000 7fcbdaffd640 10 monclient: _finish_auth 0 2024-03-29T11:21:17.974 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:17.995+0000 7fcbdaffd640 10 monclient: _check_auth_tickets 2024-03-29T11:21:17.974 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:17.995+0000 7fcbdaffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:21:17.975 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:17.995+0000 7fcbdaffd640 1 -- 172.21.15.84:0/664901756 <== mon.0 v1:172.21.15.84:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fcbe0001c50 con 0x7fcbec14ff40 2024-03-29T11:21:17.975 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:17.995+0000 7fcbdaffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:21:17.975 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:17.995+0000 7fcbdaffd640 1 -- 172.21.15.84:0/664901756 <== mon.0 v1:172.21.15.84:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4236868694 0 0) 0x7fcbe0001740 con 0x7fcbec14ff40 2024-03-29T11:21:17.975 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:17.995+0000 7fcbdaffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:17.975 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:17.995+0000 7fcbdaffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:21:17.975 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:17.995+0000 7fcbdaffd640 10 monclient: dump: 2024-03-29T11:21:17.975 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T11:21:17.975 INFO:teuthology.misc.health.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:21:17.975 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T11:18:53.864025+0000 2024-03-29T11:21:17.975 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T11:18:53.864025+0000 2024-03-29T11:21:17.975 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:17.975 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:21:17.975 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T11:21:17.975 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T11:21:17.975 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T11:21:17.976 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:21:17.976 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:17.995+0000 7fcbdbfff640 4 set_mon_vals no callback set 2024-03-29T11:21:17.978 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:17.996+0000 7fcbf0a4e640 5 monclient: authenticate success, global_id 4302 2024-03-29T11:21:17.978 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:17.996+0000 7fcbf0a4e640 10 monclient: _renew_subs 2024-03-29T11:21:17.978 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:17.996+0000 7fcbf0a4e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.84:6789/0 2024-03-29T11:21:17.978 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:17.996+0000 7fcbf0a4e640 1 -- 172.21.15.84:0/664901756 --> v1:172.21.15.84:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcbec06f1c0 con 0x7fcbec14ff40 2024-03-29T11:21:17.978 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:17.996+0000 7fcbf0a4e640 10 client.4302.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:21:17.978 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:17.996+0000 7fcbf0a4e640 10 monclient: _renew_subs 2024-03-29T11:21:17.978 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:17.996+0000 7fcbf0a4e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.84:6789/0 2024-03-29T11:21:17.978 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:17.996+0000 7fcbf0a4e640 1 -- 172.21.15.84:0/664901756 --> v1:172.21.15.84:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fcbec072530 con 0x7fcbec14ff40 2024-03-29T11:21:17.978 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:17.996+0000 7fcbf0a4e640 1 librados: init done 2024-03-29T11:21:17.980 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:17.999+0000 7fcbf0a4e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:21:17.980 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:17.999+0000 7fcbf0a4e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:21:17.980 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:17.999+0000 7fcbf0a4e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.84:6789/0 2024-03-29T11:21:17.980 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:17.999+0000 7fcbf0a4e640 1 -- 172.21.15.84:0/664901756 --> v1:172.21.15.84:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcbec06fd70 con 0x7fcbec14ff40 2024-03-29T11:21:17.980 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:17.999+0000 7fcbdaffd640 1 -- 172.21.15.84:0/664901756 <== mon.0 v1:172.21.15.84:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4033942909 0 0) 0x7fcbe0004860 con 0x7fcbec14ff40 2024-03-29T11:21:17.980 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:18.000+0000 7fcbdaffd640 1 -- 172.21.15.84:0/664901756 <== mon.0 v1:172.21.15.84:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2808139587 0 0) 0x7fcbe0084650 con 0x7fcbec14ff40 2024-03-29T11:21:17.980 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:18.000+0000 7fcbdaffd640 10 client.4302.objecter ms_dispatch 0x7fcbec000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:21:17.980 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:18.000+0000 7fcbdaffd640 3 client.4302.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:21:17.980 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:18.000+0000 7fcbdaffd640 3 client.4302.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:21:17.980 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:18.000+0000 7fcbdaffd640 20 client.4302.objecter dump_active .. 0 homeless 2024-03-29T11:21:17.981 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:18.001+0000 7fcbea59c640 10 monclient: get_auth_request con 0x7fcbcc06e040 auth_method 0 2024-03-29T11:21:17.981 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:18.001+0000 7fcbdaffd640 10 client.4302.objecter ms_handle_connect 0x7fcbcc06e040 2024-03-29T11:21:17.989 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:18.012+0000 7fcbdaffd640 1 -- 172.21.15.84:0/664901756 <== mon.0 v1:172.21.15.84:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fcbe004fd60 con 0x7fcbec14ff40 2024-03-29T11:21:17.989 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:18.012+0000 7fcbdaffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:21:17.989 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:18.012+0000 7fcbdaffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:21:18.305 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:18.329+0000 7fcbf0a4e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:21:18.306 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:18.329+0000 7fcbf0a4e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:21:18.306 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:18.329+0000 7fcbf0a4e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.84:6789/0 2024-03-29T11:21:18.306 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:18.329+0000 7fcbf0a4e640 1 -- 172.21.15.84:0/664901756 --> v1:172.21.15.84:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fcbec06f350 con 0x7fcbec14ff40 2024-03-29T11:21:18.306 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:18.330+0000 7fcbdaffd640 1 -- 172.21.15.84:0/664901756 <== mon.0 v1:172.21.15.84:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4209477549) 0x7fcbe0051e60 con 0x7fcbec14ff40 2024-03-29T11:21:18.306 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:18.330+0000 7fcbdaffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:21:18.307 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:18.330+0000 7fcbdaffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:21:18.307 INFO:teuthology.misc.health.smithi084.stdout: 2024-03-29T11:21:18.307 INFO:teuthology.misc.health.smithi084.stdout:{"epoch":10,"fsid":"2c3e002f-9eb0-4860-853b-4ef1ef562213","created":"2024-03-29T11:19:32.029169+0000","modified":"2024-03-29T11:19:34.200299+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:34.200299+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"5c39e276-2034-4716-bbc0-57a634eefa92","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"cd580e99-109f-4bf1-88c0-a5987bc8b9ed","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"11e60a5a-051d-453f-a190-66387cb35592","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"2c329fbe-bbc4-4030-bf87-d711cad04679","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f4213d4a-bd36-4433-91ae-d7e2b02481d3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"f0cf44ea-6055-48af-805b-aed858102ff9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"d895eaa0-cc0b-4ee4-9b5a-2dfcd89a60b7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"9bc5a7bc-8f52-4db9-96d6-de5e68d52eaa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:21:18.310 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:18.334+0000 7fcbf0a4e640 10 librados: watch_flush enter 2024-03-29T11:21:18.310 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:18.334+0000 7fcbf0a4e640 10 librados: watch_flush exit 2024-03-29T11:21:18.310 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:18.334+0000 7fcbf0a4e640 20 client.4302.objecter shutdown clearing up homeless session... 2024-03-29T11:21:18.310 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:18.334+0000 7fcbf0a4e640 10 client.4302.objecter successfully canceled tick 2024-03-29T11:21:18.312 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:18.336+0000 7fcbf0a4e640 1 -- 172.21.15.84:0/664901756 >> v1:172.21.15.179:6816/2981134980 conn(0x7fcbcc06e040 legacy=0x7fcbcc070500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:18.312 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:18.336+0000 7fcbf0a4e640 10 monclient: shutdown 2024-03-29T11:21:18.313 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:18.336+0000 7fcbf0a4e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:21:18.313 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:18.336+0000 7fcbf0a4e640 1 -- 172.21.15.84:0/664901756 >> v1:172.21.15.84:6789/0 conn(0x7fcbec14ff40 legacy=0x7fcbec071c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:18.313 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:18.336+0000 7fcbf0a4e640 1 -- 172.21.15.84:0/664901756 shutdown_connections 2024-03-29T11:21:18.313 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:18.336+0000 7fcbf0a4e640 1 -- 172.21.15.84:0/664901756 >> 172.21.15.84:0/664901756 conn(0x7fcbec0b35e0 msgr2=0x7fcbec0b39c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:21:18.313 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:18.336+0000 7fcbf0a4e640 1 -- 172.21.15.84:0/664901756 shutdown_connections 2024-03-29T11:21:18.313 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:18.336+0000 7fcbf0a4e640 1 -- 172.21.15.84:0/664901756 wait complete. 2024-03-29T11:21:18.313 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:18.336+0000 7fcbf0a4e640 1 librados: shutdown 2024-03-29T11:21:18.329 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:21:25.136 DEBUG:teuthology.orchestra.run.smithi084:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:21:25.286 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:25.310+0000 7fcf7c863640 10 monclient: get_monmap_and_config 2024-03-29T11:21:25.287 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:25.310+0000 7fcf7c863640 10 monclient: build_initial_monmap 2024-03-29T11:21:25.287 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:25.310+0000 7fcf7c863640 10 monclient: monmap: 2024-03-29T11:21:25.287 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-29T11:21:25.287 INFO:teuthology.misc.health.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:21:25.287 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T11:21:25.311274+0000 2024-03-29T11:21:25.287 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T11:21:25.311274+0000 2024-03-29T11:21:25.287 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-29T11:21:25.287 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:21:25.287 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-29T11:21:25.287 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.noname-c 2024-03-29T11:21:25.287 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-b 2024-03-29T11:21:25.287 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:21:25.288 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:25.311+0000 7fcf7c863640 1 Processor -- start 2024-03-29T11:21:25.288 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:25.311+0000 7fcf7c863640 1 -- start start 2024-03-29T11:21:25.288 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:25.311+0000 7fcf7c863640 10 monclient: init 2024-03-29T11:21:25.289 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:25.312+0000 7fcf7c863640 10 monclient: _reopen_session rank -1 2024-03-29T11:21:25.290 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:25.312+0000 7fcf7c863640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:21:25.290 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:25.312+0000 7fcf7c863640 10 monclient(hunting): picked mon.noname-a con 0x7fcf780b1f60 addr v1:172.21.15.84:6789/0 2024-03-29T11:21:25.290 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:25.312+0000 7fcf7c863640 10 monclient(hunting): picked mon.noname-b con 0x7fcf78071eb0 addr v1:172.21.15.84:6790/0 2024-03-29T11:21:25.290 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:25.312+0000 7fcf7c863640 10 monclient(hunting): picked mon.noname-c con 0x7fcf78143d10 addr v1:172.21.15.179:6789/0 2024-03-29T11:21:25.290 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:25.312+0000 7fcf7c863640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcf7806ef80 con 0x7fcf780b1f60 2024-03-29T11:21:25.290 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:25.312+0000 7fcf7c863640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcf7806f260 con 0x7fcf78143d10 2024-03-29T11:21:25.290 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:25.312+0000 7fcf7c863640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcf780b40c0 con 0x7fcf78071eb0 2024-03-29T11:21:25.290 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:25.312+0000 7fcf7c863640 10 monclient(hunting): _renew_subs 2024-03-29T11:21:25.290 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:25.312+0000 7fcf7c863640 10 monclient(hunting): authenticate will time out at 946.606693s 2024-03-29T11:21:25.294 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:25.318+0000 7fcf7659c640 1 --1- >> v1:172.21.15.84:6789/0 conn(0x7fcf780b1f60 0x7fcf78143600 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.84:6789/0 says I am v1:172.21.15.84:42103/0 (socket says 172.21.15.84:42103) 2024-03-29T11:21:25.294 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:25.318+0000 7fcf7659c640 1 -- 172.21.15.84:0/3052772252 learned_addr learned my addr 172.21.15.84:0/3052772252 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-29T11:21:25.295 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:25.318+0000 7fcf7559a640 1 -- 172.21.15.84:0/3052772252 <== mon.0 v1:172.21.15.84:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1845464872 0 0) 0x7fcf7806ef80 con 0x7fcf780b1f60 2024-03-29T11:21:25.295 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:25.318+0000 7fcf7559a640 10 monclient(hunting): _negotiate 2024-03-29T11:21:25.295 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:25.318+0000 7fcf7559a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:21:25.295 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:25.318+0000 7fcf7559a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:21:25.295 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:25.318+0000 7fcf7559a640 10 monclient(hunting): my global_id is 4305 2024-03-29T11:21:25.295 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:25.318+0000 7fcf7559a640 1 -- 172.21.15.84:0/3052772252 --> v1:172.21.15.84:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcf64003520 con 0x7fcf780b1f60 2024-03-29T11:21:25.296 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:25.319+0000 7fcf7559a640 1 -- 172.21.15.84:0/3052772252 <== mon.0 v1:172.21.15.84:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 848568496 0 0) 0x7fcf64003520 con 0x7fcf780b1f60 2024-03-29T11:21:25.296 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:25.319+0000 7fcf7559a640 1 -- 172.21.15.84:0/3052772252 >> v1:172.21.15.84:6790/0 conn(0x7fcf78071eb0 legacy=0x7fcf78146470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:21:25.296 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:25.319+0000 7fcf7559a640 1 -- 172.21.15.84:0/3052772252 >> v1:172.21.15.179:6789/0 conn(0x7fcf78143d10 legacy=0x7fcf78146b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:21:25.296 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:25.319+0000 7fcf7559a640 10 monclient: _finish_hunting 0 2024-03-29T11:21:25.296 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:25.319+0000 7fcf7559a640 1 monclient: found mon.noname-a 2024-03-29T11:21:25.296 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:25.319+0000 7fcf7559a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:21:25.296 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:25.319+0000 7fcf7559a640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.84:6789/0 2024-03-29T11:21:25.296 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:25.319+0000 7fcf7559a640 1 -- 172.21.15.84:0/3052772252 --> v1:172.21.15.84:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf78155d70 con 0x7fcf780b1f60 2024-03-29T11:21:25.296 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:25.319+0000 7fcf7559a640 10 monclient: _finish_auth 0 2024-03-29T11:21:25.296 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:25.319+0000 7fcf7559a640 10 monclient: _check_auth_tickets 2024-03-29T11:21:25.296 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:25.319+0000 7fcf7559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:21:25.296 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:25.319+0000 7fcf7559a640 1 -- 172.21.15.84:0/3052772252 <== mon.0 v1:172.21.15.84:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4236868694 0 0) 0x7fcf60001670 con 0x7fcf780b1f60 2024-03-29T11:21:25.297 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:25.319+0000 7fcf7559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:25.297 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:25.319+0000 7fcf7559a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:21:25.297 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:25.319+0000 7fcf7559a640 10 monclient: dump: 2024-03-29T11:21:25.297 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T11:21:25.297 INFO:teuthology.misc.health.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:21:25.297 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T11:18:53.864025+0000 2024-03-29T11:21:25.297 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T11:18:53.864025+0000 2024-03-29T11:21:25.297 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:25.297 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:21:25.297 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T11:21:25.297 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T11:21:25.297 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T11:21:25.297 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:21:25.297 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:25.319+0000 7fcf7c863640 5 monclient: authenticate success, global_id 4305 2024-03-29T11:21:25.298 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:25.319+0000 7fcf7c863640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:21:25.298 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:25.319+0000 7fcf7559a640 1 -- 172.21.15.84:0/3052772252 <== mon.0 v1:172.21.15.84:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fcf600018d0 con 0x7fcf780b1f60 2024-03-29T11:21:25.298 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:25.319+0000 7fcf7559a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:21:25.300 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:25.319+0000 7fcf7c863640 10 monclient: get_monmap_and_config success 2024-03-29T11:21:25.301 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:25.319+0000 7fcf7c863640 4 set_mon_vals no callback set 2024-03-29T11:21:25.301 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:25.319+0000 7fcf7559a640 1 -- 172.21.15.84:0/3052772252 <== mon.0 v1:172.21.15.84:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4236868694 0 0) 0x7fcf60001670 con 0x7fcf780b1f60 2024-03-29T11:21:25.301 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:25.319+0000 7fcf7559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:25.301 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:25.319+0000 7fcf7559a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:21:25.301 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:25.319+0000 7fcf7559a640 10 monclient: dump: 2024-03-29T11:21:25.301 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T11:21:25.301 INFO:teuthology.misc.health.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:21:25.301 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T11:18:53.864025+0000 2024-03-29T11:21:25.301 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T11:18:53.864025+0000 2024-03-29T11:21:25.301 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:25.301 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:21:25.301 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T11:21:25.301 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T11:21:25.301 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T11:21:25.301 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:21:25.302 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:25.320+0000 7fcf7c863640 10 monclient: shutdown 2024-03-29T11:21:25.302 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:25.320+0000 7fcf7c863640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:21:25.302 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:25.320+0000 7fcf7c863640 1 -- 172.21.15.84:0/3052772252 >> v1:172.21.15.84:6789/0 conn(0x7fcf780b1f60 legacy=0x7fcf78143600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:25.302 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:25.323+0000 7fcf7c863640 1 -- 172.21.15.84:0/3052772252 shutdown_connections 2024-03-29T11:21:25.302 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:25.323+0000 7fcf7c863640 1 -- 172.21.15.84:0/3052772252 >> 172.21.15.84:0/3052772252 conn(0x7fcf78066a00 msgr2=0x7fcf78066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:21:25.302 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:25.324+0000 7fcf7c863640 1 -- 172.21.15.84:0/3052772252 shutdown_connections 2024-03-29T11:21:25.302 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:25.324+0000 7fcf7c863640 1 -- 172.21.15.84:0/3052772252 wait complete. 2024-03-29T11:21:25.303 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:25.325+0000 7fcf7c863640 10 monclient: build_initial_monmap 2024-03-29T11:21:25.303 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:25.325+0000 7fcf7c863640 10 monclient: monmap: 2024-03-29T11:21:25.303 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-29T11:21:25.303 INFO:teuthology.misc.health.smithi084.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:21:25.303 INFO:teuthology.misc.health.smithi084.stderr:last_changed 0.000000 2024-03-29T11:21:25.303 INFO:teuthology.misc.health.smithi084.stderr:created 0.000000 2024-03-29T11:21:25.303 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-29T11:21:25.303 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:21:25.303 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-29T11:21:25.303 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.noname-b 2024-03-29T11:21:25.303 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-c 2024-03-29T11:21:25.304 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:21:25.304 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:25.325+0000 7fcf7c863640 1 librados: starting msgr at 2024-03-29T11:21:25.304 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:25.325+0000 7fcf7c863640 1 librados: starting objecter 2024-03-29T11:21:25.304 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:25.326+0000 7fcf7c863640 1 Processor -- start 2024-03-29T11:21:25.304 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:25.326+0000 7fcf7c863640 1 -- start start 2024-03-29T11:21:25.304 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:25.326+0000 7fcf7c863640 1 librados: setting wanted keys 2024-03-29T11:21:25.304 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:25.326+0000 7fcf7c863640 1 librados: calling monclient init 2024-03-29T11:21:25.304 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:25.326+0000 7fcf7c863640 10 monclient: init 2024-03-29T11:21:25.305 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:25.327+0000 7fcf7c863640 10 monclient: _reopen_session rank -1 2024-03-29T11:21:25.305 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:25.327+0000 7fcf7c863640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:21:25.305 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:25.328+0000 7fcf7c863640 10 monclient(hunting): picked mon.noname-a con 0x7fcf78143d10 addr v1:172.21.15.84:6789/0 2024-03-29T11:21:25.307 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:25.330+0000 7fcf7659c640 1 --1- >> v1:172.21.15.84:6789/0 conn(0x7fcf78143d10 0x7fcf78147cb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.84:6789/0 says I am v1:172.21.15.84:42269/0 (socket says 172.21.15.84:42269) 2024-03-29T11:21:25.308 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:25.331+0000 7fcf75d9b640 1 --1- >> v1:172.21.15.84:6790/0 conn(0x7fcf78145360 0x7fcf781457f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.84:6790/0 says I am v1:172.21.15.84:46453/0 (socket says 172.21.15.84:46453) 2024-03-29T11:21:25.308 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:25.331+0000 7fcf7c863640 10 monclient(hunting): picked mon.noname-c con 0x7fcf78145360 addr v1:172.21.15.84:6790/0 2024-03-29T11:21:25.308 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:25.331+0000 7fcf7c863640 10 monclient(hunting): picked mon.noname-b con 0x7fcf781e40a0 addr v1:172.21.15.179:6789/0 2024-03-29T11:21:25.308 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:25.331+0000 7fcf7c863640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcf78149d50 con 0x7fcf78143d10 2024-03-29T11:21:25.308 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:25.331+0000 7fcf7c863640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcf7814a030 con 0x7fcf781e40a0 2024-03-29T11:21:25.308 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:25.331+0000 7fcf7c863640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcf7814a310 con 0x7fcf78145360 2024-03-29T11:21:25.308 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:25.331+0000 7fcf7c863640 10 monclient(hunting): _renew_subs 2024-03-29T11:21:25.308 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:25.331+0000 7fcf7c863640 10 monclient(hunting): authenticate will time out at 946.626004s 2024-03-29T11:21:25.311 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:25.332+0000 7fcf7659c640 1 -- 172.21.15.84:0/3755645299 learned_addr learned my addr 172.21.15.84:0/3755645299 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-29T11:21:25.311 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:25.333+0000 7fcf6effd640 10 client.?.objecter ms_handle_connect 0x7fcf78145360 2024-03-29T11:21:25.312 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:25.333+0000 7fcf6effd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:21:25.312 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:25.333+0000 7fcf6effd640 10 client.?.objecter ms_handle_connect 0x7fcf78143d10 2024-03-29T11:21:25.312 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:25.333+0000 7fcf6effd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:21:25.312 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:25.333+0000 7fcf6effd640 1 -- 172.21.15.84:0/3755645299 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3410434876 0 0) 0x7fcf7814a310 con 0x7fcf78145360 2024-03-29T11:21:25.312 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:25.333+0000 7fcf6effd640 10 monclient(hunting): _negotiate 2024-03-29T11:21:25.312 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:25.333+0000 7fcf6effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:21:25.312 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:25.333+0000 7fcf6effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:21:25.312 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:25.334+0000 7fcf6effd640 10 monclient(hunting): my global_id is 4304 2024-03-29T11:21:25.312 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:25.334+0000 7fcf6effd640 1 -- 172.21.15.84:0/3755645299 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcf58004eb0 con 0x7fcf78145360 2024-03-29T11:21:25.312 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:25.334+0000 7fcf6effd640 1 -- 172.21.15.84:0/3755645299 <== mon.0 v1:172.21.15.84:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1683624591 0 0) 0x7fcf78149d50 con 0x7fcf78143d10 2024-03-29T11:21:25.312 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:25.334+0000 7fcf6effd640 10 monclient(hunting): _negotiate 2024-03-29T11:21:25.312 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:25.334+0000 7fcf6effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:21:25.313 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:25.334+0000 7fcf6effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:21:25.313 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:25.334+0000 7fcf6effd640 10 monclient(hunting): my global_id is 4308 2024-03-29T11:21:25.313 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:25.334+0000 7fcf6effd640 1 -- 172.21.15.84:0/3755645299 --> v1:172.21.15.84:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcf7814a310 con 0x7fcf78143d10 2024-03-29T11:21:25.313 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:25.334+0000 7fcf6effd640 1 -- 172.21.15.84:0/3755645299 <== mon.2 v1:172.21.15.84:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 126023441 0 0) 0x7fcf58004eb0 con 0x7fcf78145360 2024-03-29T11:21:25.313 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:25.334+0000 7fcf6effd640 1 -- 172.21.15.84:0/3755645299 --> v1:172.21.15.84:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fcf78149d50 con 0x7fcf78145360 2024-03-29T11:21:25.313 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:25.334+0000 7fcf6effd640 1 -- 172.21.15.84:0/3755645299 <== mon.2 v1:172.21.15.84:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4236868694 0 0) 0x7fcf680018a0 con 0x7fcf78145360 2024-03-29T11:21:25.313 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:25.334+0000 7fcf6effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:25.313 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:25.334+0000 7fcf6effd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:21:25.313 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:25.334+0000 7fcf6effd640 10 monclient(hunting): dump: 2024-03-29T11:21:25.313 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T11:21:25.313 INFO:teuthology.misc.health.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:21:25.313 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T11:18:53.864025+0000 2024-03-29T11:21:25.313 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T11:18:53.864025+0000 2024-03-29T11:21:25.314 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:25.314 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:21:25.314 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T11:21:25.314 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T11:21:25.314 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T11:21:25.314 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:21:25.314 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:25.334+0000 7fcf6effd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:21:25.314 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:25.335+0000 7fcf6effd640 1 -- 172.21.15.84:0/3755645299 <== mon.2 v1:172.21.15.84:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1312583377 0 0) 0x7fcf78149d50 con 0x7fcf78145360 2024-03-29T11:21:25.314 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:25.335+0000 7fcf6effd640 1 -- 172.21.15.84:0/3755645299 >> v1:172.21.15.179:6789/0 conn(0x7fcf781e40a0 legacy=0x7fcf781e6490 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:21:25.317 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:25.336+0000 7fcf6effd640 1 -- 172.21.15.84:0/3755645299 >> v1:172.21.15.84:6789/0 conn(0x7fcf78143d10 legacy=0x7fcf78147cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:25.317 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:25.336+0000 7fcf6effd640 10 monclient: _finish_hunting 0 2024-03-29T11:21:25.317 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:25.336+0000 7fcf6effd640 1 monclient: found mon.c 2024-03-29T11:21:25.317 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:25.336+0000 7fcf6effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:21:25.317 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:25.336+0000 7fcf6effd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-29T11:21:25.317 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:25.336+0000 7fcf6effd640 1 -- 172.21.15.84:0/3755645299 --> v1:172.21.15.84:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf78148f20 con 0x7fcf78145360 2024-03-29T11:21:25.317 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:25.336+0000 7fcf6effd640 10 monclient: _finish_auth 0 2024-03-29T11:21:25.317 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:25.336+0000 7fcf6effd640 10 monclient: _check_auth_tickets 2024-03-29T11:21:25.317 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:25.336+0000 7fcf6effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:21:25.317 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:25.337+0000 7fcf7c863640 5 monclient: authenticate success, global_id 4304 2024-03-29T11:21:25.318 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:25.337+0000 7fcf7c863640 10 monclient: _renew_subs 2024-03-29T11:21:25.318 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:25.337+0000 7fcf7c863640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-29T11:21:25.318 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:25.337+0000 7fcf7c863640 1 -- 172.21.15.84:0/3755645299 --> v1:172.21.15.84:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcf78145f00 con 0x7fcf78145360 2024-03-29T11:21:25.318 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:25.337+0000 7fcf7c863640 10 client.4304.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:21:25.318 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:25.337+0000 7fcf7c863640 10 monclient: _renew_subs 2024-03-29T11:21:25.318 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:25.337+0000 7fcf7c863640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-29T11:21:25.318 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:25.337+0000 7fcf7c863640 1 -- 172.21.15.84:0/3755645299 --> v1:172.21.15.84:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fcf78148cb0 con 0x7fcf78145360 2024-03-29T11:21:25.318 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:25.337+0000 7fcf7c863640 1 librados: init done 2024-03-29T11:21:25.318 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:25.338+0000 7fcf6effd640 1 -- 172.21.15.84:0/3755645299 <== mon.2 v1:172.21.15.84:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fcf68001a00 con 0x7fcf78145360 2024-03-29T11:21:25.318 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:25.338+0000 7fcf6effd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:21:25.318 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:25.338+0000 7fcf6effd640 1 -- 172.21.15.84:0/3755645299 <== mon.2 v1:172.21.15.84:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4236868694 0 0) 0x7fcf680037d0 con 0x7fcf78145360 2024-03-29T11:21:25.318 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:25.338+0000 7fcf6effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:25.321 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:25.338+0000 7fcf6effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:21:25.321 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:25.338+0000 7fcf6effd640 10 monclient: dump: 2024-03-29T11:21:25.321 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T11:21:25.321 INFO:teuthology.misc.health.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:21:25.321 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T11:18:53.864025+0000 2024-03-29T11:21:25.321 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T11:18:53.864025+0000 2024-03-29T11:21:25.322 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:25.322 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:21:25.322 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T11:21:25.322 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T11:21:25.322 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T11:21:25.322 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:21:25.322 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:25.339+0000 7fcf74d99640 4 set_mon_vals no callback set 2024-03-29T11:21:25.322 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:25.340+0000 7fcf7c863640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:21:25.322 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:25.340+0000 7fcf7c863640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:21:25.322 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:25.340+0000 7fcf7c863640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-29T11:21:25.322 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:25.340+0000 7fcf7c863640 1 -- 172.21.15.84:0/3755645299 --> v1:172.21.15.84:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcf5c005080 con 0x7fcf78145360 2024-03-29T11:21:25.323 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:25.342+0000 7fcf6effd640 1 -- 172.21.15.84:0/3755645299 <== mon.2 v1:172.21.15.84:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4033942909 0 0) 0x7fcf680039b0 con 0x7fcf78145360 2024-03-29T11:21:25.324 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:25.343+0000 7fcf6effd640 1 -- 172.21.15.84:0/3755645299 <== mon.2 v1:172.21.15.84:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2808139587 0 0) 0x7fcf68084a40 con 0x7fcf78145360 2024-03-29T11:21:25.324 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:25.343+0000 7fcf6effd640 10 client.4304.objecter ms_dispatch 0x7fcf78000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:21:25.324 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:25.343+0000 7fcf6effd640 3 client.4304.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:21:25.324 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:25.343+0000 7fcf6effd640 3 client.4304.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:21:25.324 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:25.343+0000 7fcf6effd640 20 client.4304.objecter dump_active .. 0 homeless 2024-03-29T11:21:25.324 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:25.344+0000 7fcf7659c640 10 monclient: get_auth_request con 0x7fcf5806dce0 auth_method 0 2024-03-29T11:21:25.324 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:25.344+0000 7fcf6effd640 10 client.4304.objecter ms_handle_connect 0x7fcf5806dce0 2024-03-29T11:21:25.326 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:25.350+0000 7fcf6effd640 1 -- 172.21.15.84:0/3755645299 <== mon.2 v1:172.21.15.84:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fcf68050150 con 0x7fcf78145360 2024-03-29T11:21:25.327 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:25.350+0000 7fcf6effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:21:25.327 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:25.350+0000 7fcf6effd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:21:25.642 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:25.665+0000 7fcf7c863640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:21:25.642 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:25.665+0000 7fcf7c863640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:21:25.642 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:25.665+0000 7fcf7c863640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-29T11:21:25.642 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:25.665+0000 7fcf7c863640 1 -- 172.21.15.84:0/3755645299 --> v1:172.21.15.84:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fcf5c005330 con 0x7fcf78145360 2024-03-29T11:21:25.646 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:25.669+0000 7fcf6effd640 1 -- 172.21.15.84:0/3755645299 <== mon.2 v1:172.21.15.84:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4209477549) 0x7fcf68052250 con 0x7fcf78145360 2024-03-29T11:21:25.646 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:25.669+0000 7fcf6effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:21:25.646 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:25.669+0000 7fcf6effd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:21:25.646 INFO:teuthology.misc.health.smithi084.stdout: 2024-03-29T11:21:25.646 INFO:teuthology.misc.health.smithi084.stdout:{"epoch":10,"fsid":"2c3e002f-9eb0-4860-853b-4ef1ef562213","created":"2024-03-29T11:19:32.029169+0000","modified":"2024-03-29T11:19:34.200299+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:34.200299+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"5c39e276-2034-4716-bbc0-57a634eefa92","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"cd580e99-109f-4bf1-88c0-a5987bc8b9ed","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"11e60a5a-051d-453f-a190-66387cb35592","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"2c329fbe-bbc4-4030-bf87-d711cad04679","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f4213d4a-bd36-4433-91ae-d7e2b02481d3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"f0cf44ea-6055-48af-805b-aed858102ff9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"d895eaa0-cc0b-4ee4-9b5a-2dfcd89a60b7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"9bc5a7bc-8f52-4db9-96d6-de5e68d52eaa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:21:25.649 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:25.673+0000 7fcf7c863640 10 librados: watch_flush enter 2024-03-29T11:21:25.649 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:25.673+0000 7fcf7c863640 10 librados: watch_flush exit 2024-03-29T11:21:25.649 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:25.673+0000 7fcf7c863640 20 client.4304.objecter shutdown clearing up homeless session... 2024-03-29T11:21:25.649 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:25.673+0000 7fcf7c863640 10 client.4304.objecter successfully canceled tick 2024-03-29T11:21:25.649 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:25.673+0000 7fcf7c863640 1 -- 172.21.15.84:0/3755645299 >> v1:172.21.15.179:6816/2981134980 conn(0x7fcf5806dce0 legacy=0x7fcf580701a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:25.649 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:25.673+0000 7fcf7c863640 10 monclient: shutdown 2024-03-29T11:21:25.650 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:25.673+0000 7fcf7c863640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:21:25.650 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:25.673+0000 7fcf7c863640 1 -- 172.21.15.84:0/3755645299 >> v1:172.21.15.84:6790/0 conn(0x7fcf78145360 legacy=0x7fcf781457f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:25.650 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:25.674+0000 7fcf7c863640 1 -- 172.21.15.84:0/3755645299 shutdown_connections 2024-03-29T11:21:25.650 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:25.674+0000 7fcf7c863640 1 -- 172.21.15.84:0/3755645299 >> 172.21.15.84:0/3755645299 conn(0x7fcf78066a00 msgr2=0x7fcf78142f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:21:25.652 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:25.676+0000 7fcf7c863640 1 -- 172.21.15.84:0/3755645299 shutdown_connections 2024-03-29T11:21:25.652 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:25.676+0000 7fcf7c863640 1 -- 172.21.15.84:0/3755645299 wait complete. 2024-03-29T11:21:25.653 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:25.676+0000 7fcf7c863640 1 librados: shutdown 2024-03-29T11:21:25.669 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:21:32.476 DEBUG:teuthology.orchestra.run.smithi084:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:21:32.624 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:32.647+0000 7f0bb30d8640 10 monclient: get_monmap_and_config 2024-03-29T11:21:32.624 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:32.648+0000 7f0bb30d8640 10 monclient: build_initial_monmap 2024-03-29T11:21:32.624 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:32.648+0000 7f0bb30d8640 10 monclient: monmap: 2024-03-29T11:21:32.625 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-29T11:21:32.625 INFO:teuthology.misc.health.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:21:32.625 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T11:21:32.648934+0000 2024-03-29T11:21:32.625 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T11:21:32.648934+0000 2024-03-29T11:21:32.625 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-29T11:21:32.625 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:21:32.625 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-29T11:21:32.625 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.noname-c 2024-03-29T11:21:32.625 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-b 2024-03-29T11:21:32.625 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:21:32.626 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:32.649+0000 7f0bb30d8640 1 Processor -- start 2024-03-29T11:21:32.630 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:32.650+0000 7f0bb30d8640 1 -- start start 2024-03-29T11:21:32.630 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:32.650+0000 7f0bb30d8640 10 monclient: init 2024-03-29T11:21:32.630 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:32.651+0000 7f0bb30d8640 10 monclient: _reopen_session rank -1 2024-03-29T11:21:32.630 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:32.651+0000 7f0bb30d8640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:21:32.630 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:32.651+0000 7f0bb30d8640 10 monclient(hunting): picked mon.noname-b con 0x7f0bac0b4820 addr v1:172.21.15.84:6790/0 2024-03-29T11:21:32.630 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:32.651+0000 7f0bb30d8640 10 monclient(hunting): picked mon.noname-a con 0x7f0bac0a9ce0 addr v1:172.21.15.84:6789/0 2024-03-29T11:21:32.630 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:32.651+0000 7f0bb30d8640 10 monclient(hunting): picked mon.noname-c con 0x7f0bac0b3cd0 addr v1:172.21.15.179:6789/0 2024-03-29T11:21:32.630 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:32.651+0000 7f0bb30d8640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0bac0b4180 con 0x7f0bac0a9ce0 2024-03-29T11:21:32.631 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:32.651+0000 7f0bb30d8640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0bac0aaf60 con 0x7f0bac0b3cd0 2024-03-29T11:21:32.631 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:32.651+0000 7f0bb30d8640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0bac13c200 con 0x7f0bac0b4820 2024-03-29T11:21:32.631 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:32.651+0000 7f0bb30d8640 10 monclient(hunting): _renew_subs 2024-03-29T11:21:32.631 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:32.651+0000 7f0bb30d8640 10 monclient(hunting): authenticate will time out at 953.945856s 2024-03-29T11:21:32.637 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:32.655+0000 7f0bb28d7640 1 --1- >> v1:172.21.15.179:6789/0 conn(0x7f0bac0b3cd0 0x7f0bac14d7a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.179:6789/0 says I am v1:172.21.15.84:55713/0 (socket says 172.21.15.84:55713) 2024-03-29T11:21:32.637 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:32.655+0000 7f0bb28d7640 1 -- 172.21.15.84:0/209123599 learned_addr learned my addr 172.21.15.84:0/209123599 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-29T11:21:32.637 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:32.658+0000 7f0bb10d4640 1 -- 172.21.15.84:0/209123599 <== mon.1 v1:172.21.15.179:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 281006588 0 0) 0x7f0bac0aaf60 con 0x7f0bac0b3cd0 2024-03-29T11:21:32.637 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:32.658+0000 7f0bb10d4640 10 monclient(hunting): _negotiate 2024-03-29T11:21:32.637 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:32.658+0000 7f0bb10d4640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:21:32.637 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:32.658+0000 7f0bb10d4640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:21:32.637 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:32.658+0000 7f0bb10d4640 10 monclient(hunting): my global_id is 4276 2024-03-29T11:21:32.637 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:32.658+0000 7f0bb10d4640 1 -- 172.21.15.84:0/209123599 --> v1:172.21.15.179:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0b98003520 con 0x7f0bac0b3cd0 2024-03-29T11:21:32.637 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:32.659+0000 7f0bb10d4640 1 -- 172.21.15.84:0/209123599 <== mon.1 v1:172.21.15.179:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2038921480 0 0) 0x7f0b98003520 con 0x7f0bac0b3cd0 2024-03-29T11:21:32.637 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:32.659+0000 7f0bb10d4640 1 -- 172.21.15.84:0/209123599 >> v1:172.21.15.84:6790/0 conn(0x7f0bac0b4820 legacy=0x7f0bac13baf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:21:32.638 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:32.659+0000 7f0bb10d4640 1 -- 172.21.15.84:0/209123599 >> v1:172.21.15.84:6789/0 conn(0x7f0bac0a9ce0 legacy=0x7f0bac145770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:21:32.638 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:32.659+0000 7f0bb10d4640 10 monclient: _finish_hunting 0 2024-03-29T11:21:32.638 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:32.659+0000 7f0bb10d4640 1 monclient: found mon.noname-c 2024-03-29T11:21:32.638 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:32.659+0000 7f0bb10d4640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:21:32.638 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:32.659+0000 7f0bb10d4640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.179:6789/0 2024-03-29T11:21:32.638 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:32.659+0000 7f0bb10d4640 1 -- 172.21.15.84:0/209123599 --> v1:172.21.15.179:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0bac145e80 con 0x7f0bac0b3cd0 2024-03-29T11:21:32.638 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:32.659+0000 7f0bb10d4640 10 monclient: _finish_auth 0 2024-03-29T11:21:32.638 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:32.659+0000 7f0bb10d4640 10 monclient: _check_auth_tickets 2024-03-29T11:21:32.638 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:32.659+0000 7f0bb10d4640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:21:32.638 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:32.659+0000 7f0bb30d8640 5 monclient: authenticate success, global_id 4276 2024-03-29T11:21:32.638 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:32.659+0000 7f0bb30d8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:21:32.638 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:32.659+0000 7f0bb10d4640 1 -- 172.21.15.84:0/209123599 <== mon.1 v1:172.21.15.179:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4236868694 0 0) 0x7f0ba8001690 con 0x7f0bac0b3cd0 2024-03-29T11:21:32.638 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:32.659+0000 7f0bb10d4640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:32.638 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:32.659+0000 7f0bb10d4640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:21:32.639 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:32.659+0000 7f0bb10d4640 10 monclient: dump: 2024-03-29T11:21:32.639 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T11:21:32.639 INFO:teuthology.misc.health.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:21:32.639 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T11:18:53.864025+0000 2024-03-29T11:21:32.639 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T11:18:53.864025+0000 2024-03-29T11:21:32.639 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:32.639 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:21:32.639 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T11:21:32.639 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T11:21:32.639 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T11:21:32.639 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:21:32.639 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:32.659+0000 7f0bb30d8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:21:32.639 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:32.659+0000 7f0bb10d4640 1 -- 172.21.15.84:0/209123599 <== mon.1 v1:172.21.15.179:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0ba8001860 con 0x7f0bac0b3cd0 2024-03-29T11:21:32.639 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:32.659+0000 7f0bb10d4640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:21:32.639 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:32.659+0000 7f0bb10d4640 1 -- 172.21.15.84:0/209123599 <== mon.1 v1:172.21.15.179:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4236868694 0 0) 0x7f0ba8001cb0 con 0x7f0bac0b3cd0 2024-03-29T11:21:32.640 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:32.659+0000 7f0bb10d4640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:32.640 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:32.659+0000 7f0bb10d4640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:21:32.640 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:32.659+0000 7f0bb10d4640 10 monclient: dump: 2024-03-29T11:21:32.640 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T11:21:32.640 INFO:teuthology.misc.health.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:21:32.640 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T11:18:53.864025+0000 2024-03-29T11:21:32.640 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T11:18:53.864025+0000 2024-03-29T11:21:32.640 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:32.640 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:21:32.640 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T11:21:32.640 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T11:21:32.641 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T11:21:32.641 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:21:32.641 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:32.660+0000 7f0bb30d8640 10 monclient: get_monmap_and_config success 2024-03-29T11:21:32.641 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:32.660+0000 7f0bb30d8640 4 set_mon_vals no callback set 2024-03-29T11:21:32.641 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:32.660+0000 7f0bb30d8640 10 monclient: shutdown 2024-03-29T11:21:32.641 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:32.660+0000 7f0bb30d8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:21:32.641 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:32.660+0000 7f0bb30d8640 1 -- 172.21.15.84:0/209123599 >> v1:172.21.15.179:6789/0 conn(0x7f0bac0b3cd0 legacy=0x7f0bac14d7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:32.641 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:32.661+0000 7f0bb30d8640 1 -- 172.21.15.84:0/209123599 shutdown_connections 2024-03-29T11:21:32.641 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:32.661+0000 7f0bb30d8640 1 -- 172.21.15.84:0/209123599 >> 172.21.15.84:0/209123599 conn(0x7f0bac06d070 msgr2=0x7f0bac066a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:21:32.642 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:32.661+0000 7f0bb30d8640 1 -- 172.21.15.84:0/209123599 shutdown_connections 2024-03-29T11:21:32.642 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:32.665+0000 7f0bb30d8640 1 -- 172.21.15.84:0/209123599 wait complete. 2024-03-29T11:21:32.643 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:32.666+0000 7f0bb30d8640 10 monclient: build_initial_monmap 2024-03-29T11:21:32.643 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:32.666+0000 7f0bb30d8640 10 monclient: monmap: 2024-03-29T11:21:32.643 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-29T11:21:32.643 INFO:teuthology.misc.health.smithi084.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:21:32.643 INFO:teuthology.misc.health.smithi084.stderr:last_changed 0.000000 2024-03-29T11:21:32.643 INFO:teuthology.misc.health.smithi084.stderr:created 0.000000 2024-03-29T11:21:32.643 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-29T11:21:32.643 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:21:32.644 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-29T11:21:32.644 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.noname-b 2024-03-29T11:21:32.644 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-c 2024-03-29T11:21:32.644 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:21:32.644 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:32.666+0000 7f0bb30d8640 1 librados: starting msgr at 2024-03-29T11:21:32.644 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:32.666+0000 7f0bb30d8640 1 librados: starting objecter 2024-03-29T11:21:32.644 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:32.666+0000 7f0bb30d8640 1 Processor -- start 2024-03-29T11:21:32.647 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:32.667+0000 7f0bb30d8640 1 -- start start 2024-03-29T11:21:32.647 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:32.667+0000 7f0bb30d8640 1 librados: setting wanted keys 2024-03-29T11:21:32.647 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:32.667+0000 7f0bb30d8640 1 librados: calling monclient init 2024-03-29T11:21:32.647 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:32.667+0000 7f0bb30d8640 10 monclient: init 2024-03-29T11:21:32.647 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:32.668+0000 7f0bb30d8640 10 monclient: _reopen_session rank -1 2024-03-29T11:21:32.647 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:32.668+0000 7f0bb30d8640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:21:32.647 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:32.668+0000 7f0bb30d8640 10 monclient(hunting): picked mon.noname-c con 0x7f0bac0a9ce0 addr v1:172.21.15.84:6790/0 2024-03-29T11:21:32.647 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:32.668+0000 7f0bb30d8640 10 monclient(hunting): picked mon.noname-b con 0x7f0bac0b3cd0 addr v1:172.21.15.179:6789/0 2024-03-29T11:21:32.647 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:32.668+0000 7f0bb30d8640 10 monclient(hunting): picked mon.noname-a con 0x7f0bac0b4820 addr v1:172.21.15.84:6789/0 2024-03-29T11:21:32.647 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:32.668+0000 7f0bb30d8640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0bac071610 con 0x7f0bac0b4820 2024-03-29T11:21:32.648 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:32.668+0000 7f0bb30d8640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0bac1db940 con 0x7f0bac0b3cd0 2024-03-29T11:21:32.648 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:32.668+0000 7f0bb30d8640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0bac071860 con 0x7f0bac0a9ce0 2024-03-29T11:21:32.648 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:32.668+0000 7f0bb30d8640 10 monclient(hunting): _renew_subs 2024-03-29T11:21:32.648 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:32.668+0000 7f0bb30d8640 10 monclient(hunting): authenticate will time out at 953.962207s 2024-03-29T11:21:32.648 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:32.671+0000 7f0bb20d6640 1 --1- >> v1:172.21.15.84:6790/0 conn(0x7f0bac0a9ce0 0x7f0bac06f390 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.84:6790/0 says I am v1:172.21.15.84:42887/0 (socket says 172.21.15.84:42887) 2024-03-29T11:21:32.649 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:32.671+0000 7f0bb20d6640 1 -- 172.21.15.84:0/3560912194 learned_addr learned my addr 172.21.15.84:0/3560912194 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-29T11:21:32.651 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:32.674+0000 7f0ba2ffd640 10 client.?.objecter ms_handle_connect 0x7f0bac0a9ce0 2024-03-29T11:21:32.651 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:32.674+0000 7f0ba2ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:21:32.651 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:32.674+0000 7f0ba2ffd640 1 -- 172.21.15.84:0/3560912194 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1161981479 0 0) 0x7f0bac071860 con 0x7f0bac0a9ce0 2024-03-29T11:21:32.651 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:32.674+0000 7f0ba2ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:21:32.652 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:32.675+0000 7f0ba2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:21:32.652 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:32.675+0000 7f0ba2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:21:32.652 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:32.675+0000 7f0ba2ffd640 10 monclient(hunting): my global_id is 4307 2024-03-29T11:21:32.652 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:32.675+0000 7f0ba2ffd640 1 -- 172.21.15.84:0/3560912194 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0b90004eb0 con 0x7f0bac0a9ce0 2024-03-29T11:21:32.652 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:32.675+0000 7f0ba2ffd640 10 client.?.objecter ms_handle_connect 0x7f0bac0b4820 2024-03-29T11:21:32.652 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:32.675+0000 7f0ba2ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:21:32.652 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:32.675+0000 7f0ba2ffd640 1 -- 172.21.15.84:0/3560912194 <== mon.0 v1:172.21.15.84:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3996514707 0 0) 0x7f0bac071610 con 0x7f0bac0b4820 2024-03-29T11:21:32.653 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:32.675+0000 7f0ba2ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:21:32.653 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:32.675+0000 7f0ba2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:21:32.653 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:32.675+0000 7f0ba2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:21:32.653 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:32.675+0000 7f0ba2ffd640 10 monclient(hunting): my global_id is 4311 2024-03-29T11:21:32.653 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:32.675+0000 7f0ba2ffd640 1 -- 172.21.15.84:0/3560912194 --> v1:172.21.15.84:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0bac071860 con 0x7f0bac0b4820 2024-03-29T11:21:32.653 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:32.676+0000 7f0ba2ffd640 1 -- 172.21.15.84:0/3560912194 <== mon.0 v1:172.21.15.84:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1206716700 0 0) 0x7f0bac071860 con 0x7f0bac0b4820 2024-03-29T11:21:32.653 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:32.676+0000 7f0ba2ffd640 1 -- 172.21.15.84:0/3560912194 --> v1:172.21.15.84:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0b90005490 con 0x7f0bac0b4820 2024-03-29T11:21:32.653 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:32.676+0000 7f0ba2ffd640 1 -- 172.21.15.84:0/3560912194 <== mon.0 v1:172.21.15.84:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4236868694 0 0) 0x7f0ba8001b70 con 0x7f0bac0b4820 2024-03-29T11:21:32.653 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:32.676+0000 7f0ba2ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:32.653 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:32.676+0000 7f0ba2ffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:21:32.653 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:32.676+0000 7f0ba2ffd640 10 monclient(hunting): dump: 2024-03-29T11:21:32.653 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T11:21:32.653 INFO:teuthology.misc.health.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:21:32.654 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T11:18:53.864025+0000 2024-03-29T11:21:32.654 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T11:18:53.864025+0000 2024-03-29T11:21:32.654 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:32.654 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:21:32.654 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T11:21:32.654 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T11:21:32.654 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T11:21:32.654 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:21:32.654 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:32.676+0000 7f0ba2ffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:21:32.656 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:32.676+0000 7f0ba2ffd640 1 -- 172.21.15.84:0/3560912194 <== mon.0 v1:172.21.15.84:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 318279358 0 0) 0x7f0b90005490 con 0x7f0bac0b4820 2024-03-29T11:21:32.656 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:32.677+0000 7f0ba2ffd640 1 -- 172.21.15.84:0/3560912194 >> v1:172.21.15.84:6790/0 conn(0x7f0bac0a9ce0 legacy=0x7f0bac06f390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:32.656 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:32.677+0000 7f0ba2ffd640 1 -- 172.21.15.84:0/3560912194 >> v1:172.21.15.179:6789/0 conn(0x7f0bac0b3cd0 legacy=0x7f0bac1db230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:32.656 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:32.677+0000 7f0ba2ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:21:32.656 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:32.677+0000 7f0ba2ffd640 1 monclient: found mon.a 2024-03-29T11:21:32.656 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:32.677+0000 7f0ba2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:21:32.656 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:32.677+0000 7f0ba2ffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.84:6789/0 2024-03-29T11:21:32.656 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:32.677+0000 7f0ba2ffd640 1 -- 172.21.15.84:0/3560912194 --> v1:172.21.15.84:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0bac144030 con 0x7f0bac0b4820 2024-03-29T11:21:32.656 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:32.677+0000 7f0ba2ffd640 10 monclient: _finish_auth 0 2024-03-29T11:21:32.656 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:32.677+0000 7f0ba2ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:21:32.657 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:32.677+0000 7f0ba2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:21:32.657 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:32.677+0000 7f0bb30d8640 5 monclient: authenticate success, global_id 4311 2024-03-29T11:21:32.657 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:32.677+0000 7f0bb30d8640 10 monclient: _renew_subs 2024-03-29T11:21:32.657 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:32.677+0000 7f0bb30d8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.84:6789/0 2024-03-29T11:21:32.657 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:32.677+0000 7f0bb30d8640 1 -- 172.21.15.84:0/3560912194 --> v1:172.21.15.84:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0bac144280 con 0x7f0bac0b4820 2024-03-29T11:21:32.657 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:32.677+0000 7f0bb30d8640 10 client.4311.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:21:32.657 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:32.677+0000 7f0bb30d8640 10 monclient: _renew_subs 2024-03-29T11:21:32.657 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:32.677+0000 7f0bb30d8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.84:6789/0 2024-03-29T11:21:32.657 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:32.677+0000 7f0bb30d8640 1 -- 172.21.15.84:0/3560912194 --> v1:172.21.15.84:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f0bac070460 con 0x7f0bac0b4820 2024-03-29T11:21:32.657 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:32.677+0000 7f0bb30d8640 1 librados: init done 2024-03-29T11:21:32.657 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:32.678+0000 7f0bb30d8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:21:32.657 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:32.678+0000 7f0bb30d8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:21:32.658 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:32.679+0000 7f0bb30d8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.84:6789/0 2024-03-29T11:21:32.658 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:32.679+0000 7f0bb30d8640 1 -- 172.21.15.84:0/3560912194 --> v1:172.21.15.84:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0bac06faa0 con 0x7f0bac0b4820 2024-03-29T11:21:32.658 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:32.679+0000 7f0ba2ffd640 1 -- 172.21.15.84:0/3560912194 <== mon.0 v1:172.21.15.84:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0ba8001d50 con 0x7f0bac0b4820 2024-03-29T11:21:32.658 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:32.679+0000 7f0ba2ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:21:32.658 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:32.679+0000 7f0ba2ffd640 1 -- 172.21.15.84:0/3560912194 <== mon.0 v1:172.21.15.84:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4236868694 0 0) 0x7f0ba8001b70 con 0x7f0bac0b4820 2024-03-29T11:21:32.658 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:32.679+0000 7f0ba2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:32.658 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:32.679+0000 7f0ba2ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:21:32.658 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:32.679+0000 7f0ba2ffd640 10 monclient: dump: 2024-03-29T11:21:32.658 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T11:21:32.658 INFO:teuthology.misc.health.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:21:32.659 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T11:18:53.864025+0000 2024-03-29T11:21:32.659 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T11:18:53.864025+0000 2024-03-29T11:21:32.659 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:32.659 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:21:32.659 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T11:21:32.659 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T11:21:32.659 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T11:21:32.659 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:21:32.659 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:32.679+0000 7f0bb08d3640 4 set_mon_vals no callback set 2024-03-29T11:21:32.660 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:32.681+0000 7f0ba2ffd640 1 -- 172.21.15.84:0/3560912194 <== mon.0 v1:172.21.15.84:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4033942909 0 0) 0x7f0ba801aa60 con 0x7f0bac0b4820 2024-03-29T11:21:32.660 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:32.682+0000 7f0bb20d6640 10 monclient: get_auth_request con 0x7f0b9006e160 auth_method 0 2024-03-29T11:21:32.660 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:32.683+0000 7f0ba2ffd640 1 -- 172.21.15.84:0/3560912194 <== mon.0 v1:172.21.15.84:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2808139587 0 0) 0x7f0ba8054470 con 0x7f0bac0b4820 2024-03-29T11:21:32.660 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:32.683+0000 7f0ba2ffd640 10 client.4311.objecter ms_dispatch 0x7f0bac000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:21:32.660 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:32.683+0000 7f0ba2ffd640 3 client.4311.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:21:32.660 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:32.683+0000 7f0ba2ffd640 3 client.4311.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:21:32.661 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:32.683+0000 7f0ba2ffd640 20 client.4311.objecter dump_active .. 0 homeless 2024-03-29T11:21:32.661 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:32.683+0000 7f0ba2ffd640 10 client.4311.objecter ms_handle_connect 0x7f0b9006e160 2024-03-29T11:21:32.664 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:32.687+0000 7f0ba2ffd640 1 -- 172.21.15.84:0/3560912194 <== mon.0 v1:172.21.15.84:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f0ba8050710 con 0x7f0bac0b4820 2024-03-29T11:21:32.664 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:32.688+0000 7f0ba2ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:21:32.664 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:32.688+0000 7f0ba2ffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:21:32.991 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:33.015+0000 7f0bb30d8640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:21:32.991 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:33.015+0000 7f0bb30d8640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:21:32.992 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:33.015+0000 7f0bb30d8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.84:6789/0 2024-03-29T11:21:32.992 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:33.015+0000 7f0bb30d8640 1 -- 172.21.15.84:0/3560912194 --> v1:172.21.15.84:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0bac04f730 con 0x7f0bac0b4820 2024-03-29T11:21:32.992 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:33.016+0000 7f0ba2ffd640 1 -- 172.21.15.84:0/3560912194 <== mon.0 v1:172.21.15.84:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4209477549) 0x7f0ba80508b0 con 0x7f0bac0b4820 2024-03-29T11:21:32.992 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:33.016+0000 7f0ba2ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:21:32.992 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:33.016+0000 7f0ba2ffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:21:32.992 INFO:teuthology.misc.health.smithi084.stdout: 2024-03-29T11:21:32.993 INFO:teuthology.misc.health.smithi084.stdout:{"epoch":10,"fsid":"2c3e002f-9eb0-4860-853b-4ef1ef562213","created":"2024-03-29T11:19:32.029169+0000","modified":"2024-03-29T11:19:34.200299+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:34.200299+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"5c39e276-2034-4716-bbc0-57a634eefa92","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"cd580e99-109f-4bf1-88c0-a5987bc8b9ed","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"11e60a5a-051d-453f-a190-66387cb35592","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"2c329fbe-bbc4-4030-bf87-d711cad04679","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f4213d4a-bd36-4433-91ae-d7e2b02481d3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"f0cf44ea-6055-48af-805b-aed858102ff9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"d895eaa0-cc0b-4ee4-9b5a-2dfcd89a60b7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"9bc5a7bc-8f52-4db9-96d6-de5e68d52eaa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:21:32.996 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:33.019+0000 7f0bb30d8640 10 librados: watch_flush enter 2024-03-29T11:21:32.996 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:33.019+0000 7f0bb30d8640 10 librados: watch_flush exit 2024-03-29T11:21:32.996 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:33.019+0000 7f0bb30d8640 20 client.4311.objecter shutdown clearing up homeless session... 2024-03-29T11:21:32.996 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:33.019+0000 7f0bb30d8640 10 client.4311.objecter successfully canceled tick 2024-03-29T11:21:32.996 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:33.019+0000 7f0bb30d8640 1 -- 172.21.15.84:0/3560912194 >> v1:172.21.15.179:6816/2981134980 conn(0x7f0b9006e160 legacy=0x7f0b90070620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:32.996 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:33.019+0000 7f0bb30d8640 10 monclient: shutdown 2024-03-29T11:21:32.996 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:33.019+0000 7f0bb30d8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:21:32.996 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:33.019+0000 7f0bb30d8640 1 -- 172.21.15.84:0/3560912194 >> v1:172.21.15.84:6789/0 conn(0x7f0bac0b4820 legacy=0x7f0bac13a480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:32.996 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:33.019+0000 7f0bb30d8640 1 -- 172.21.15.84:0/3560912194 shutdown_connections 2024-03-29T11:21:32.996 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:33.019+0000 7f0bb30d8640 1 -- 172.21.15.84:0/3560912194 >> 172.21.15.84:0/3560912194 conn(0x7f0bac06d070 msgr2=0x7f0bac0b8220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:21:32.997 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:33.020+0000 7f0bb30d8640 1 -- 172.21.15.84:0/3560912194 shutdown_connections 2024-03-29T11:21:32.997 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:33.020+0000 7f0bb30d8640 1 -- 172.21.15.84:0/3560912194 wait complete. 2024-03-29T11:21:32.997 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:33.020+0000 7f0bb30d8640 1 librados: shutdown 2024-03-29T11:21:33.015 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:21:39.826 DEBUG:teuthology.orchestra.run.smithi084:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:21:39.975 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:39.999+0000 7f785202d640 10 monclient: get_monmap_and_config 2024-03-29T11:21:39.975 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:39.999+0000 7f785202d640 10 monclient: build_initial_monmap 2024-03-29T11:21:39.975 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:39.999+0000 7f785202d640 10 monclient: monmap: 2024-03-29T11:21:39.976 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-29T11:21:39.976 INFO:teuthology.misc.health.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:21:39.976 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T11:21:39.999968+0000 2024-03-29T11:21:39.976 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T11:21:39.999968+0000 2024-03-29T11:21:39.976 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-29T11:21:39.976 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:21:39.976 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-29T11:21:39.976 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.noname-c 2024-03-29T11:21:39.976 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-b 2024-03-29T11:21:39.976 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:21:39.978 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:39.999+0000 7f785202d640 1 Processor -- start 2024-03-29T11:21:39.978 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:40.000+0000 7f785202d640 1 -- start start 2024-03-29T11:21:39.978 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:40.000+0000 7f785202d640 10 monclient: init 2024-03-29T11:21:39.978 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:40.001+0000 7f785202d640 10 monclient: _reopen_session rank -1 2024-03-29T11:21:39.978 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:40.001+0000 7f785202d640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:21:39.978 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:40.001+0000 7f785202d640 10 monclient(hunting): picked mon.noname-a con 0x7f784c0ac310 addr v1:172.21.15.84:6789/0 2024-03-29T11:21:39.978 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:40.001+0000 7f785202d640 10 monclient(hunting): picked mon.noname-b con 0x7f784c0ae460 addr v1:172.21.15.84:6790/0 2024-03-29T11:21:39.978 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:40.001+0000 7f785202d640 10 monclient(hunting): picked mon.noname-c con 0x7f784c13d270 addr v1:172.21.15.179:6789/0 2024-03-29T11:21:39.978 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:40.001+0000 7f785202d640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f784c0add30 con 0x7f784c0ac310 2024-03-29T11:21:39.978 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:40.001+0000 7f785202d640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f784c06df00 con 0x7f784c13d270 2024-03-29T11:21:39.978 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:40.001+0000 7f785202d640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f784c0ac6f0 con 0x7f784c0ae460 2024-03-29T11:21:39.978 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:40.001+0000 7f785202d640 10 monclient(hunting): _renew_subs 2024-03-29T11:21:39.978 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:40.001+0000 7f785202d640 10 monclient(hunting): authenticate will time out at 961.295251s 2024-03-29T11:21:39.981 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:40.005+0000 7f785102b640 1 --1- >> v1:172.21.15.84:6789/0 conn(0x7f784c0ac310 0x7f784c1403c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.84:6789/0 says I am v1:172.21.15.84:46787/0 (socket says 172.21.15.84:46787) 2024-03-29T11:21:39.981 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:40.005+0000 7f785102b640 1 -- 172.21.15.84:0/3538934780 learned_addr learned my addr 172.21.15.84:0/3538934780 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-29T11:21:39.983 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:40.006+0000 7f784bfff640 1 -- 172.21.15.84:0/3538934780 <== mon.0 v1:172.21.15.84:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3123454975 0 0) 0x7f784c0add30 con 0x7f784c0ac310 2024-03-29T11:21:39.983 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:40.006+0000 7f784bfff640 10 monclient(hunting): _negotiate 2024-03-29T11:21:39.983 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:40.006+0000 7f784bfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:21:39.983 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:40.006+0000 7f784bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:21:39.983 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:40.006+0000 7f784bfff640 10 monclient(hunting): my global_id is 4314 2024-03-29T11:21:39.983 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:40.006+0000 7f784bfff640 1 -- 172.21.15.84:0/3538934780 --> v1:172.21.15.84:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7834003520 con 0x7f784c0ac310 2024-03-29T11:21:39.983 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:40.006+0000 7f784bfff640 1 -- 172.21.15.84:0/3538934780 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1846064859 0 0) 0x7f784c0ac6f0 con 0x7f784c0ae460 2024-03-29T11:21:39.983 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:40.007+0000 7f784bfff640 10 monclient(hunting): _negotiate 2024-03-29T11:21:39.988 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:40.007+0000 7f784bfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:21:39.988 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:40.008+0000 7f784bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:21:39.988 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:40.008+0000 7f784bfff640 10 monclient(hunting): my global_id is 4310 2024-03-29T11:21:39.988 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:40.008+0000 7f784bfff640 1 -- 172.21.15.84:0/3538934780 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f784c0add30 con 0x7f784c0ae460 2024-03-29T11:21:39.988 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:40.008+0000 7f784bfff640 1 -- 172.21.15.84:0/3538934780 <== mon.0 v1:172.21.15.84:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3856287696 0 0) 0x7f7834003520 con 0x7f784c0ac310 2024-03-29T11:21:39.988 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:40.008+0000 7f784bfff640 1 -- 172.21.15.84:0/3538934780 >> v1:172.21.15.84:6790/0 conn(0x7f784c0ae460 legacy=0x7f784c13cb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:39.988 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:40.008+0000 7f784bfff640 1 -- 172.21.15.84:0/3538934780 >> v1:172.21.15.179:6789/0 conn(0x7f784c13d270 legacy=0x7f784c14b4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:21:39.988 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:40.008+0000 7f784bfff640 10 monclient: _finish_hunting 0 2024-03-29T11:21:39.988 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:40.008+0000 7f784bfff640 1 monclient: found mon.noname-a 2024-03-29T11:21:39.988 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:40.008+0000 7f784bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:21:39.988 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:40.008+0000 7f784bfff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.84:6789/0 2024-03-29T11:21:39.988 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:40.008+0000 7f784bfff640 1 -- 172.21.15.84:0/3538934780 --> v1:172.21.15.84:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f784c06dc20 con 0x7f784c0ac310 2024-03-29T11:21:39.988 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:40.008+0000 7f784bfff640 10 monclient: _finish_auth 0 2024-03-29T11:21:39.989 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:40.008+0000 7f784bfff640 10 monclient: _check_auth_tickets 2024-03-29T11:21:39.989 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:40.008+0000 7f784bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:21:39.989 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:40.008+0000 7f784bfff640 1 -- 172.21.15.84:0/3538934780 <== mon.0 v1:172.21.15.84:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4236868694 0 0) 0x7f7844001a10 con 0x7f784c0ac310 2024-03-29T11:21:39.989 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:40.008+0000 7f784bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:39.989 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:40.008+0000 7f784bfff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:21:39.989 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:40.008+0000 7f784bfff640 10 monclient: dump: 2024-03-29T11:21:39.989 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T11:21:39.989 INFO:teuthology.misc.health.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:21:39.989 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T11:18:53.864025+0000 2024-03-29T11:21:39.989 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T11:18:53.864025+0000 2024-03-29T11:21:39.989 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:39.989 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:21:39.989 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T11:21:39.989 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T11:21:39.990 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T11:21:39.990 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:21:39.990 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:40.010+0000 7f785202d640 5 monclient: authenticate success, global_id 4314 2024-03-29T11:21:39.990 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:40.010+0000 7f785202d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:21:39.991 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:40.012+0000 7f784bfff640 1 -- 172.21.15.84:0/3538934780 <== mon.0 v1:172.21.15.84:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7844001b70 con 0x7f784c0ac310 2024-03-29T11:21:39.991 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:40.012+0000 7f784bfff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:21:39.991 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:40.012+0000 7f785202d640 10 monclient: get_monmap_and_config success 2024-03-29T11:21:39.991 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:40.012+0000 7f785202d640 4 set_mon_vals no callback set 2024-03-29T11:21:39.991 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:40.013+0000 7f785202d640 10 monclient: shutdown 2024-03-29T11:21:39.991 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:40.013+0000 7f785202d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:21:39.991 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:40.013+0000 7f785202d640 1 -- 172.21.15.84:0/3538934780 >> v1:172.21.15.84:6789/0 conn(0x7f784c0ac310 legacy=0x7f784c1403c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:39.991 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:40.014+0000 7f785202d640 1 -- 172.21.15.84:0/3538934780 shutdown_connections 2024-03-29T11:21:39.992 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:40.014+0000 7f785202d640 1 -- 172.21.15.84:0/3538934780 >> 172.21.15.84:0/3538934780 conn(0x7f784c0b3a30 msgr2=0x7f784c0b3e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:21:39.994 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:40.018+0000 7f785202d640 1 -- 172.21.15.84:0/3538934780 shutdown_connections 2024-03-29T11:21:39.994 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:40.018+0000 7f785202d640 1 -- 172.21.15.84:0/3538934780 wait complete. 2024-03-29T11:21:39.996 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:40.020+0000 7f785202d640 10 monclient: build_initial_monmap 2024-03-29T11:21:39.996 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:40.020+0000 7f785202d640 10 monclient: monmap: 2024-03-29T11:21:39.996 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-29T11:21:39.996 INFO:teuthology.misc.health.smithi084.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:21:39.996 INFO:teuthology.misc.health.smithi084.stderr:last_changed 0.000000 2024-03-29T11:21:39.996 INFO:teuthology.misc.health.smithi084.stderr:created 0.000000 2024-03-29T11:21:39.997 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-29T11:21:39.997 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:21:39.997 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-29T11:21:39.997 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.noname-b 2024-03-29T11:21:39.997 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-c 2024-03-29T11:21:39.997 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:21:39.999 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:40.020+0000 7f785202d640 1 librados: starting msgr at 2024-03-29T11:21:39.999 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:40.020+0000 7f785202d640 1 librados: starting objecter 2024-03-29T11:21:39.999 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:40.020+0000 7f785202d640 1 Processor -- start 2024-03-29T11:21:39.999 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:40.021+0000 7f785202d640 1 -- start start 2024-03-29T11:21:39.999 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:40.021+0000 7f785202d640 1 librados: setting wanted keys 2024-03-29T11:21:39.999 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:40.021+0000 7f785202d640 1 librados: calling monclient init 2024-03-29T11:21:39.999 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:40.021+0000 7f785202d640 10 monclient: init 2024-03-29T11:21:39.999 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:40.021+0000 7f785202d640 10 monclient: _reopen_session rank -1 2024-03-29T11:21:39.999 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:40.021+0000 7f785202d640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:21:39.999 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:40.021+0000 7f785202d640 10 monclient(hunting): picked mon.noname-c con 0x7f784c0ac310 addr v1:172.21.15.84:6790/0 2024-03-29T11:21:39.999 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:40.022+0000 7f785202d640 10 monclient(hunting): picked mon.noname-b con 0x7f784c0ae460 addr v1:172.21.15.179:6789/0 2024-03-29T11:21:39.999 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:40.022+0000 7f785202d640 10 monclient(hunting): picked mon.noname-a con 0x7f784c13d270 addr v1:172.21.15.84:6789/0 2024-03-29T11:21:40.000 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:40.022+0000 7f785202d640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f784c141720 con 0x7f784c13d270 2024-03-29T11:21:40.000 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:40.022+0000 7f785202d640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f784c1d6180 con 0x7f784c0ae460 2024-03-29T11:21:40.000 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:40.022+0000 7f785202d640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f784c13e1d0 con 0x7f784c0ac310 2024-03-29T11:21:40.000 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:40.022+0000 7f785202d640 10 monclient(hunting): _renew_subs 2024-03-29T11:21:40.000 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:40.022+0000 7f785202d640 10 monclient(hunting): authenticate will time out at 961.316446s 2024-03-29T11:21:40.004 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:40.027+0000 7f785082a640 1 --1- >> v1:172.21.15.179:6789/0 conn(0x7f784c0ae460 0x7f784c13ecb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.179:6789/0 says I am v1:172.21.15.84:48357/0 (socket says 172.21.15.84:48357) 2024-03-29T11:21:40.004 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:40.027+0000 7f785082a640 1 -- 172.21.15.84:0/1292452655 learned_addr learned my addr 172.21.15.84:0/1292452655 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-29T11:21:40.004 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:40.028+0000 7f7849ffb640 10 client.?.objecter ms_handle_connect 0x7f784c0ae460 2024-03-29T11:21:40.004 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:40.028+0000 7f7849ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:21:40.005 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:40.028+0000 7f7849ffb640 10 client.?.objecter ms_handle_connect 0x7f784c13d270 2024-03-29T11:21:40.005 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:40.028+0000 7f7849ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:21:40.005 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:40.028+0000 7f7849ffb640 1 -- 172.21.15.84:0/1292452655 <== mon.1 v1:172.21.15.179:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 455427429 0 0) 0x7f784c1d6180 con 0x7f784c0ae460 2024-03-29T11:21:40.005 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:40.028+0000 7f7849ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:21:40.005 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:40.028+0000 7f7849ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:21:40.005 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:40.028+0000 7f7849ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:21:40.005 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:40.028+0000 7f7849ffb640 10 monclient(hunting): my global_id is 4279 2024-03-29T11:21:40.005 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:40.028+0000 7f7849ffb640 1 -- 172.21.15.84:0/1292452655 --> v1:172.21.15.179:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7838004eb0 con 0x7f784c0ae460 2024-03-29T11:21:40.005 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:40.028+0000 7f7849ffb640 1 -- 172.21.15.84:0/1292452655 <== mon.0 v1:172.21.15.84:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1832443311 0 0) 0x7f784c141720 con 0x7f784c13d270 2024-03-29T11:21:40.005 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:40.028+0000 7f7849ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:21:40.005 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:40.028+0000 7f7849ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:21:40.005 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:40.028+0000 7f7849ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:21:40.006 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:40.028+0000 7f7849ffb640 10 monclient(hunting): my global_id is 4317 2024-03-29T11:21:40.006 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:40.028+0000 7f7849ffb640 1 -- 172.21.15.84:0/1292452655 --> v1:172.21.15.84:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f784c1d6180 con 0x7f784c13d270 2024-03-29T11:21:40.009 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:40.029+0000 7f7849ffb640 1 -- 172.21.15.84:0/1292452655 <== mon.1 v1:172.21.15.179:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2824434165 0 0) 0x7f7838004eb0 con 0x7f784c0ae460 2024-03-29T11:21:40.009 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:40.029+0000 7f7849ffb640 1 -- 172.21.15.84:0/1292452655 --> v1:172.21.15.179:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f784c141720 con 0x7f784c0ae460 2024-03-29T11:21:40.010 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:40.029+0000 7f7849ffb640 1 -- 172.21.15.84:0/1292452655 <== mon.1 v1:172.21.15.179:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4236868694 0 0) 0x7f783c001a50 con 0x7f784c0ae460 2024-03-29T11:21:40.010 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:40.029+0000 7f7849ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:40.010 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:40.029+0000 7f7849ffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:21:40.010 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:40.029+0000 7f7849ffb640 10 monclient(hunting): dump: 2024-03-29T11:21:40.010 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T11:21:40.010 INFO:teuthology.misc.health.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:21:40.010 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T11:18:53.864025+0000 2024-03-29T11:21:40.010 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T11:18:53.864025+0000 2024-03-29T11:21:40.010 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:40.010 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:21:40.010 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T11:21:40.010 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T11:21:40.011 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T11:21:40.011 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:21:40.011 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:40.029+0000 7f7849ffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:21:40.011 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:40.029+0000 7f7849ffb640 1 -- 172.21.15.84:0/1292452655 <== mon.0 v1:172.21.15.84:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3583793648 0 0) 0x7f784c1d6180 con 0x7f784c13d270 2024-03-29T11:21:40.011 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:40.029+0000 7f7849ffb640 1 -- 172.21.15.84:0/1292452655 --> v1:172.21.15.84:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7838004eb0 con 0x7f784c13d270 2024-03-29T11:21:40.011 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:40.029+0000 7f7849ffb640 1 -- 172.21.15.84:0/1292452655 <== mon.0 v1:172.21.15.84:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4236868694 0 0) 0x7f7840001710 con 0x7f784c13d270 2024-03-29T11:21:40.011 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:40.029+0000 7f7849ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:40.011 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:40.029+0000 7f7849ffb640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:21:40.011 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:40.029+0000 7f7849ffb640 10 monclient(hunting): dump: 2024-03-29T11:21:40.011 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T11:21:40.011 INFO:teuthology.misc.health.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:21:40.011 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T11:18:53.864025+0000 2024-03-29T11:21:40.011 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T11:18:53.864025+0000 2024-03-29T11:21:40.011 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:40.012 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:21:40.012 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T11:21:40.012 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T11:21:40.012 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T11:21:40.012 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:21:40.012 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:40.029+0000 7f7849ffb640 1 -- 172.21.15.84:0/1292452655 <== mon.1 v1:172.21.15.179:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3716948741 0 0) 0x7f784c141720 con 0x7f784c0ae460 2024-03-29T11:21:40.012 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:40.029+0000 7f7849ffb640 1 -- 172.21.15.84:0/1292452655 >> v1:172.21.15.84:6790/0 conn(0x7f784c0ac310 legacy=0x7f784c1d66a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:21:40.012 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:40.029+0000 7f7849ffb640 1 -- 172.21.15.84:0/1292452655 >> v1:172.21.15.84:6789/0 conn(0x7f784c13d270 legacy=0x7f784c133960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:40.012 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:40.029+0000 7f7849ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:21:40.012 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:40.030+0000 7f7849ffb640 1 monclient: found mon.b 2024-03-29T11:21:40.012 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:40.030+0000 7f7849ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:21:40.013 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:40.030+0000 7f7849ffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-29T11:21:40.013 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:40.030+0000 7f7849ffb640 1 -- 172.21.15.84:0/1292452655 --> v1:172.21.15.179:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f784c1d6db0 con 0x7f784c0ae460 2024-03-29T11:21:40.013 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:40.030+0000 7f7849ffb640 10 monclient: _finish_auth 0 2024-03-29T11:21:40.013 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:40.030+0000 7f7849ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:21:40.013 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:40.030+0000 7f7849ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:21:40.013 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:40.030+0000 7f785202d640 5 monclient: authenticate success, global_id 4279 2024-03-29T11:21:40.013 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:40.030+0000 7f785202d640 10 monclient: _renew_subs 2024-03-29T11:21:40.013 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:40.030+0000 7f785202d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-29T11:21:40.013 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:40.030+0000 7f785202d640 1 -- 172.21.15.84:0/1292452655 --> v1:172.21.15.179:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f784c0a94e0 con 0x7f784c0ae460 2024-03-29T11:21:40.013 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:40.030+0000 7f785202d640 10 client.4279.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:21:40.013 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:40.031+0000 7f785202d640 10 monclient: _renew_subs 2024-03-29T11:21:40.013 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:40.031+0000 7f7849ffb640 1 -- 172.21.15.84:0/1292452655 <== mon.1 v1:172.21.15.179:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f783c001c90 con 0x7f784c0ae460 2024-03-29T11:21:40.013 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:40.031+0000 7f785202d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-29T11:21:40.014 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:40.031+0000 7f785202d640 1 -- 172.21.15.84:0/1292452655 --> v1:172.21.15.179:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f784c13f5a0 con 0x7f784c0ae460 2024-03-29T11:21:40.014 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:40.031+0000 7f785202d640 1 librados: init done 2024-03-29T11:21:40.014 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:40.032+0000 7f7849ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:21:40.014 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:40.032+0000 7f7849ffb640 1 -- 172.21.15.84:0/1292452655 <== mon.1 v1:172.21.15.179:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4236868694 0 0) 0x7f783c003a60 con 0x7f784c0ae460 2024-03-29T11:21:40.014 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:40.032+0000 7f7849ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:40.014 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:40.032+0000 7f7849ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:21:40.014 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:40.032+0000 7f784b7fe640 4 set_mon_vals no callback set 2024-03-29T11:21:40.014 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:40.032+0000 7f7849ffb640 10 monclient: dump: 2024-03-29T11:21:40.014 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T11:21:40.014 INFO:teuthology.misc.health.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:21:40.014 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T11:18:53.864025+0000 2024-03-29T11:21:40.014 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T11:18:53.864025+0000 2024-03-29T11:21:40.014 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:40.015 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:21:40.015 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T11:21:40.015 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T11:21:40.015 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T11:21:40.015 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:21:40.016 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:40.035+0000 7f7849ffb640 1 -- 172.21.15.84:0/1292452655 <== mon.1 v1:172.21.15.179:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4033942909 0 0) 0x7f783c003bc0 con 0x7f784c0ae460 2024-03-29T11:21:40.016 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:40.035+0000 7f7849ffb640 1 -- 172.21.15.84:0/1292452655 <== mon.1 v1:172.21.15.179:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2808139587 0 0) 0x7f783c084b40 con 0x7f784c0ae460 2024-03-29T11:21:40.016 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:40.035+0000 7f7849ffb640 10 client.4279.objecter ms_dispatch 0x7f784c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:21:40.016 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:40.035+0000 7f7849ffb640 3 client.4279.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:21:40.016 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:40.035+0000 7f7849ffb640 3 client.4279.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:21:40.017 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:40.036+0000 7f7849ffb640 20 client.4279.objecter dump_active .. 0 homeless 2024-03-29T11:21:40.017 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:40.036+0000 7f78337fe640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:21:40.017 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:40.036+0000 7f78337fe640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:21:40.017 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:40.036+0000 7f78337fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-29T11:21:40.017 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:40.036+0000 7f78337fe640 1 -- 172.21.15.84:0/1292452655 --> v1:172.21.15.179:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f782c005080 con 0x7f784c0ae460 2024-03-29T11:21:40.017 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:40.037+0000 7f785102b640 10 monclient: get_auth_request con 0x7f783806ded0 auth_method 0 2024-03-29T11:21:40.017 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:40.038+0000 7f7849ffb640 10 client.4279.objecter ms_handle_connect 0x7f783806ded0 2024-03-29T11:21:40.024 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:40.047+0000 7f7849ffb640 1 -- 172.21.15.84:0/1292452655 <== mon.1 v1:172.21.15.179:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f783c050420 con 0x7f784c0ae460 2024-03-29T11:21:40.024 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:40.047+0000 7f7849ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:21:40.024 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:40.047+0000 7f7849ffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:21:40.349 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:40.373+0000 7f78337fe640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:21:40.349 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:40.373+0000 7f78337fe640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:21:40.349 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:40.373+0000 7f78337fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-29T11:21:40.349 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:40.373+0000 7f78337fe640 1 -- 172.21.15.84:0/1292452655 --> v1:172.21.15.179:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f782c005330 con 0x7f784c0ae460 2024-03-29T11:21:40.353 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:40.377+0000 7f7849ffb640 1 -- 172.21.15.84:0/1292452655 <== mon.1 v1:172.21.15.179:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4209477549) 0x7f783c052350 con 0x7f784c0ae460 2024-03-29T11:21:40.353 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:40.377+0000 7f7849ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:21:40.353 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:40.377+0000 7f7849ffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:21:40.353 INFO:teuthology.misc.health.smithi084.stdout: 2024-03-29T11:21:40.354 INFO:teuthology.misc.health.smithi084.stdout:{"epoch":10,"fsid":"2c3e002f-9eb0-4860-853b-4ef1ef562213","created":"2024-03-29T11:19:32.029169+0000","modified":"2024-03-29T11:19:34.200299+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:34.200299+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"5c39e276-2034-4716-bbc0-57a634eefa92","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"cd580e99-109f-4bf1-88c0-a5987bc8b9ed","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"11e60a5a-051d-453f-a190-66387cb35592","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"2c329fbe-bbc4-4030-bf87-d711cad04679","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f4213d4a-bd36-4433-91ae-d7e2b02481d3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"f0cf44ea-6055-48af-805b-aed858102ff9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"d895eaa0-cc0b-4ee4-9b5a-2dfcd89a60b7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"9bc5a7bc-8f52-4db9-96d6-de5e68d52eaa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:21:40.360 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:40.383+0000 7f78337fe640 10 librados: watch_flush enter 2024-03-29T11:21:40.360 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:40.384+0000 7f78337fe640 10 librados: watch_flush exit 2024-03-29T11:21:40.361 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:40.384+0000 7f78337fe640 20 client.4279.objecter shutdown clearing up homeless session... 2024-03-29T11:21:40.361 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:40.384+0000 7f78337fe640 10 client.4279.objecter successfully canceled tick 2024-03-29T11:21:40.361 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:40.384+0000 7f78337fe640 1 -- 172.21.15.84:0/1292452655 >> v1:172.21.15.179:6816/2981134980 conn(0x7f783806ded0 legacy=0x7f7838070390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:40.361 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:40.384+0000 7f78337fe640 10 monclient: shutdown 2024-03-29T11:21:40.361 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:40.384+0000 7f78337fe640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:21:40.361 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:40.384+0000 7f78337fe640 1 -- 172.21.15.84:0/1292452655 >> v1:172.21.15.179:6789/0 conn(0x7f784c0ae460 legacy=0x7f784c13ecb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:40.361 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:40.384+0000 7f78337fe640 1 -- 172.21.15.84:0/1292452655 shutdown_connections 2024-03-29T11:21:40.361 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:40.384+0000 7f78337fe640 1 -- 172.21.15.84:0/1292452655 >> 172.21.15.84:0/1292452655 conn(0x7f784c0b3a30 msgr2=0x7f784c069b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:21:40.362 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:40.385+0000 7f78337fe640 1 -- 172.21.15.84:0/1292452655 shutdown_connections 2024-03-29T11:21:40.362 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:40.385+0000 7f78337fe640 1 -- 172.21.15.84:0/1292452655 wait complete. 2024-03-29T11:21:40.365 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:40.386+0000 7f78337fe640 1 librados: shutdown 2024-03-29T11:21:40.383 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:21:47.190 DEBUG:teuthology.orchestra.run.smithi084:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:21:47.340 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:47.361+0000 7f5d6fe05640 10 monclient: get_monmap_and_config 2024-03-29T11:21:47.340 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:47.361+0000 7f5d6fe05640 10 monclient: build_initial_monmap 2024-03-29T11:21:47.340 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:47.361+0000 7f5d6fe05640 10 monclient: monmap: 2024-03-29T11:21:47.340 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-29T11:21:47.340 INFO:teuthology.misc.health.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:21:47.340 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T11:21:47.363575+0000 2024-03-29T11:21:47.340 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T11:21:47.363575+0000 2024-03-29T11:21:47.340 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-29T11:21:47.340 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:21:47.340 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-29T11:21:47.340 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.noname-c 2024-03-29T11:21:47.340 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-b 2024-03-29T11:21:47.341 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:21:47.343 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:47.363+0000 7f5d6fe05640 1 Processor -- start 2024-03-29T11:21:47.343 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:47.363+0000 7f5d6fe05640 1 -- start start 2024-03-29T11:21:47.343 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:47.363+0000 7f5d6fe05640 10 monclient: init 2024-03-29T11:21:47.343 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:47.364+0000 7f5d6fe05640 10 monclient: _reopen_session rank -1 2024-03-29T11:21:47.343 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:47.364+0000 7f5d6fe05640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:21:47.343 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:47.364+0000 7f5d6fe05640 10 monclient(hunting): picked mon.noname-b con 0x7f5d680a9930 addr v1:172.21.15.84:6790/0 2024-03-29T11:21:47.343 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:47.364+0000 7f5d6fe05640 10 monclient(hunting): picked mon.noname-c con 0x7f5d68149dc0 addr v1:172.21.15.179:6789/0 2024-03-29T11:21:47.343 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:47.364+0000 7f5d6fe05640 10 monclient(hunting): picked mon.noname-a con 0x7f5d6814ff40 addr v1:172.21.15.84:6789/0 2024-03-29T11:21:47.343 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:47.364+0000 7f5d6fe05640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5d680aae60 con 0x7f5d6814ff40 2024-03-29T11:21:47.343 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:47.364+0000 7f5d6fe05640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5d680b4310 con 0x7f5d68149dc0 2024-03-29T11:21:47.343 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:47.364+0000 7f5d6fe05640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5d680afdd0 con 0x7f5d680a9930 2024-03-29T11:21:47.343 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:47.364+0000 7f5d6fe05640 10 monclient(hunting): _renew_subs 2024-03-29T11:21:47.344 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:47.364+0000 7f5d6fe05640 10 monclient(hunting): authenticate will time out at 968.659846s 2024-03-29T11:21:47.348 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:47.370+0000 7f5d6d3a0640 1 --1- >> v1:172.21.15.179:6789/0 conn(0x7f5d68149dc0 0x7f5d6814a240 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.179:6789/0 says I am v1:172.21.15.84:44263/0 (socket says 172.21.15.84:44263) 2024-03-29T11:21:47.348 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:47.370+0000 7f5d6d3a0640 1 -- 172.21.15.84:0/3080278398 learned_addr learned my addr 172.21.15.84:0/3080278398 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-29T11:21:47.352 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:47.374+0000 7f5d6cb9f640 1 -- 172.21.15.84:0/3080278398 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1777385174 0 0) 0x7f5d680afdd0 con 0x7f5d680a9930 2024-03-29T11:21:47.356 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:47.374+0000 7f5d6cb9f640 10 monclient(hunting): _negotiate 2024-03-29T11:21:47.356 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:47.374+0000 7f5d6cb9f640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:21:47.357 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:47.374+0000 7f5d6cb9f640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:21:47.357 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:47.374+0000 7f5d6cb9f640 10 monclient(hunting): my global_id is 4313 2024-03-29T11:21:47.357 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:47.374+0000 7f5d6cb9f640 1 -- 172.21.15.84:0/3080278398 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5d54003520 con 0x7f5d680a9930 2024-03-29T11:21:47.357 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:47.375+0000 7f5d6cb9f640 1 -- 172.21.15.84:0/3080278398 <== mon.2 v1:172.21.15.84:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1744682753 0 0) 0x7f5d54003520 con 0x7f5d680a9930 2024-03-29T11:21:47.357 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:47.375+0000 7f5d6cb9f640 1 -- 172.21.15.84:0/3080278398 >> v1:172.21.15.179:6789/0 conn(0x7f5d68149dc0 legacy=0x7f5d6814a240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:47.357 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:47.375+0000 7f5d6cb9f640 1 -- 172.21.15.84:0/3080278398 >> v1:172.21.15.84:6789/0 conn(0x7f5d6814ff40 legacy=0x7f5d68152330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:21:47.357 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:47.375+0000 7f5d6cb9f640 10 monclient: _finish_hunting 0 2024-03-29T11:21:47.357 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:47.375+0000 7f5d6cb9f640 1 monclient: found mon.noname-b 2024-03-29T11:21:47.357 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:47.375+0000 7f5d6cb9f640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:21:47.357 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:47.375+0000 7f5d6cb9f640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.84:6790/0 2024-03-29T11:21:47.357 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:47.375+0000 7f5d6cb9f640 1 -- 172.21.15.84:0/3080278398 --> v1:172.21.15.84:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d6814a950 con 0x7f5d680a9930 2024-03-29T11:21:47.357 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:47.375+0000 7f5d6cb9f640 10 monclient: _finish_auth 0 2024-03-29T11:21:47.357 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:47.375+0000 7f5d6cb9f640 10 monclient: _check_auth_tickets 2024-03-29T11:21:47.357 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:47.375+0000 7f5d6cb9f640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:21:47.358 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:47.375+0000 7f5d6cb9f640 1 -- 172.21.15.84:0/3080278398 <== mon.2 v1:172.21.15.84:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4236868694 0 0) 0x7f5d60001a10 con 0x7f5d680a9930 2024-03-29T11:21:47.358 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:47.375+0000 7f5d6cb9f640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:47.358 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:47.375+0000 7f5d6cb9f640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:21:47.358 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:47.375+0000 7f5d6cb9f640 10 monclient: dump: 2024-03-29T11:21:47.358 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T11:21:47.358 INFO:teuthology.misc.health.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:21:47.358 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T11:18:53.864025+0000 2024-03-29T11:21:47.358 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T11:18:53.864025+0000 2024-03-29T11:21:47.358 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:47.358 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:21:47.358 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T11:21:47.358 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T11:21:47.358 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T11:21:47.358 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:21:47.359 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:47.375+0000 7f5d6fe05640 5 monclient: authenticate success, global_id 4313 2024-03-29T11:21:47.359 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:47.375+0000 7f5d6fe05640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:21:47.359 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:47.375+0000 7f5d6cb9f640 1 -- 172.21.15.84:0/3080278398 <== mon.2 v1:172.21.15.84:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5d60001be0 con 0x7f5d680a9930 2024-03-29T11:21:47.359 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:47.375+0000 7f5d6cb9f640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:21:47.359 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:47.375+0000 7f5d6cb9f640 1 -- 172.21.15.84:0/3080278398 <== mon.2 v1:172.21.15.84:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4236868694 0 0) 0x7f5d60001ff0 con 0x7f5d680a9930 2024-03-29T11:21:47.359 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:47.375+0000 7f5d6cb9f640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:47.359 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:47.375+0000 7f5d6cb9f640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:21:47.359 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:47.375+0000 7f5d6cb9f640 10 monclient: dump: 2024-03-29T11:21:47.359 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T11:21:47.359 INFO:teuthology.misc.health.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:21:47.359 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T11:18:53.864025+0000 2024-03-29T11:21:47.359 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T11:18:53.864025+0000 2024-03-29T11:21:47.359 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:47.359 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:21:47.359 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T11:21:47.360 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T11:21:47.360 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T11:21:47.360 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:21:47.360 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:47.375+0000 7f5d6fe05640 10 monclient: get_monmap_and_config success 2024-03-29T11:21:47.360 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:47.375+0000 7f5d6fe05640 4 set_mon_vals no callback set 2024-03-29T11:21:47.360 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:47.376+0000 7f5d6fe05640 10 monclient: shutdown 2024-03-29T11:21:47.360 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:47.376+0000 7f5d6fe05640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:21:47.360 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:47.376+0000 7f5d6fe05640 1 -- 172.21.15.84:0/3080278398 >> v1:172.21.15.84:6790/0 conn(0x7f5d680a9930 legacy=0x7f5d68143a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:47.360 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:47.376+0000 7f5d6fe05640 1 -- 172.21.15.84:0/3080278398 shutdown_connections 2024-03-29T11:21:47.360 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:47.376+0000 7f5d6fe05640 1 -- 172.21.15.84:0/3080278398 >> 172.21.15.84:0/3080278398 conn(0x7f5d680b35e0 msgr2=0x7f5d680b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:21:47.360 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:47.376+0000 7f5d6fe05640 1 -- 172.21.15.84:0/3080278398 shutdown_connections 2024-03-29T11:21:47.360 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:47.376+0000 7f5d6fe05640 1 -- 172.21.15.84:0/3080278398 wait complete. 2024-03-29T11:21:47.361 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:47.378+0000 7f5d6fe05640 10 monclient: build_initial_monmap 2024-03-29T11:21:47.361 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:47.378+0000 7f5d6fe05640 10 monclient: monmap: 2024-03-29T11:21:47.361 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-29T11:21:47.361 INFO:teuthology.misc.health.smithi084.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:21:47.361 INFO:teuthology.misc.health.smithi084.stderr:last_changed 0.000000 2024-03-29T11:21:47.361 INFO:teuthology.misc.health.smithi084.stderr:created 0.000000 2024-03-29T11:21:47.361 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-29T11:21:47.361 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:21:47.361 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-29T11:21:47.361 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.noname-b 2024-03-29T11:21:47.361 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-c 2024-03-29T11:21:47.362 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:21:47.362 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:47.378+0000 7f5d6fe05640 1 librados: starting msgr at 2024-03-29T11:21:47.362 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:47.379+0000 7f5d6fe05640 1 librados: starting objecter 2024-03-29T11:21:47.362 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:47.379+0000 7f5d6fe05640 1 Processor -- start 2024-03-29T11:21:47.362 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:47.379+0000 7f5d6fe05640 1 -- start start 2024-03-29T11:21:47.362 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:47.379+0000 7f5d6fe05640 1 librados: setting wanted keys 2024-03-29T11:21:47.362 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:47.379+0000 7f5d6fe05640 1 librados: calling monclient init 2024-03-29T11:21:47.363 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:47.379+0000 7f5d6fe05640 10 monclient: init 2024-03-29T11:21:47.363 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:47.380+0000 7f5d6fe05640 10 monclient: _reopen_session rank -1 2024-03-29T11:21:47.363 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:47.380+0000 7f5d6fe05640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:21:47.363 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:47.380+0000 7f5d6fe05640 10 monclient(hunting): picked mon.noname-b con 0x7f5d680a9930 addr v1:172.21.15.179:6789/0 2024-03-29T11:21:47.363 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:47.380+0000 7f5d6fe05640 10 monclient(hunting): picked mon.noname-c con 0x7f5d68149dc0 addr v1:172.21.15.84:6790/0 2024-03-29T11:21:47.363 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:47.380+0000 7f5d6fe05640 10 monclient(hunting): picked mon.noname-a con 0x7f5d6814ff40 addr v1:172.21.15.84:6789/0 2024-03-29T11:21:47.363 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:47.380+0000 7f5d6fe05640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5d6806fa90 con 0x7f5d6814ff40 2024-03-29T11:21:47.363 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:47.380+0000 7f5d6fe05640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5d6813f680 con 0x7f5d680a9930 2024-03-29T11:21:47.363 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:47.380+0000 7f5d6fe05640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5d6806f690 con 0x7f5d68149dc0 2024-03-29T11:21:47.363 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:47.380+0000 7f5d6fe05640 10 monclient(hunting): _renew_subs 2024-03-29T11:21:47.363 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:47.380+0000 7f5d6fe05640 10 monclient(hunting): authenticate will time out at 968.675211s 2024-03-29T11:21:47.364 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:47.384+0000 7f5d6dba1640 1 --1- >> v1:172.21.15.179:6789/0 conn(0x7f5d680a9930 0x7f5d680718c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.179:6789/0 says I am v1:172.21.15.84:44359/0 (socket says 172.21.15.84:44359) 2024-03-29T11:21:47.364 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:47.384+0000 7f5d6dba1640 1 -- 172.21.15.84:0/3946127605 learned_addr learned my addr 172.21.15.84:0/3946127605 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-29T11:21:47.364 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:47.386+0000 7f5d5e7fc640 10 client.?.objecter ms_handle_connect 0x7f5d680a9930 2024-03-29T11:21:47.364 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:47.386+0000 7f5d5e7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:21:47.364 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:47.386+0000 7f5d5e7fc640 1 -- 172.21.15.84:0/3946127605 <== mon.1 v1:172.21.15.179:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 743146821 0 0) 0x7f5d6813f680 con 0x7f5d680a9930 2024-03-29T11:21:47.364 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:47.386+0000 7f5d5e7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:21:47.364 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:47.386+0000 7f5d5e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:21:47.364 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:47.386+0000 7f5d5e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:21:47.364 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:47.386+0000 7f5d5e7fc640 10 monclient(hunting): my global_id is 4285 2024-03-29T11:21:47.364 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:47.386+0000 7f5d5e7fc640 1 -- 172.21.15.84:0/3946127605 --> v1:172.21.15.179:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5d4c004eb0 con 0x7f5d680a9930 2024-03-29T11:21:47.366 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:47.389+0000 7f5d5e7fc640 1 -- 172.21.15.84:0/3946127605 <== mon.1 v1:172.21.15.179:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1309395653 0 0) 0x7f5d4c004eb0 con 0x7f5d680a9930 2024-03-29T11:21:47.366 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:47.389+0000 7f5d5e7fc640 1 -- 172.21.15.84:0/3946127605 --> v1:172.21.15.179:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5d6813f680 con 0x7f5d680a9930 2024-03-29T11:21:47.367 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:47.389+0000 7f5d5e7fc640 1 -- 172.21.15.84:0/3946127605 <== mon.1 v1:172.21.15.179:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4236868694 0 0) 0x7f5d60001fb0 con 0x7f5d680a9930 2024-03-29T11:21:47.367 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:47.389+0000 7f5d5e7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:47.367 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:47.389+0000 7f5d5e7fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:21:47.367 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:47.389+0000 7f5d5e7fc640 10 monclient(hunting): dump: 2024-03-29T11:21:47.367 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T11:21:47.367 INFO:teuthology.misc.health.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:21:47.367 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T11:18:53.864025+0000 2024-03-29T11:21:47.367 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T11:18:53.864025+0000 2024-03-29T11:21:47.367 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:47.367 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:21:47.367 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T11:21:47.367 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T11:21:47.367 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T11:21:47.367 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:21:47.368 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:47.389+0000 7f5d5e7fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:21:47.368 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:47.389+0000 7f5d5e7fc640 10 client.?.objecter ms_handle_connect 0x7f5d6814ff40 2024-03-29T11:21:47.368 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:47.389+0000 7f5d5e7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:21:47.368 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:47.391+0000 7f5d5e7fc640 1 -- 172.21.15.84:0/3946127605 <== mon.1 v1:172.21.15.179:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2012044641 0 0) 0x7f5d6813f680 con 0x7f5d680a9930 2024-03-29T11:21:47.369 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:47.391+0000 7f5d5e7fc640 1 -- 172.21.15.84:0/3946127605 >> v1:172.21.15.84:6790/0 conn(0x7f5d68149dc0 legacy=0x7f5d68071fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:47.369 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:47.391+0000 7f5d5e7fc640 1 -- 172.21.15.84:0/3946127605 >> v1:172.21.15.84:6789/0 conn(0x7f5d6814ff40 legacy=0x7f5d681e5f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:47.369 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:47.391+0000 7f5d5e7fc640 10 monclient: _finish_hunting 0 2024-03-29T11:21:47.369 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:47.391+0000 7f5d5e7fc640 1 monclient: found mon.b 2024-03-29T11:21:47.369 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:47.391+0000 7f5d5e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:21:47.369 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:47.391+0000 7f5d5e7fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-29T11:21:47.369 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:47.391+0000 7f5d5e7fc640 1 -- 172.21.15.84:0/3946127605 --> v1:172.21.15.179:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d6806ee40 con 0x7f5d680a9930 2024-03-29T11:21:47.369 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:47.391+0000 7f5d5e7fc640 10 monclient: _finish_auth 0 2024-03-29T11:21:47.369 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:47.391+0000 7f5d5e7fc640 10 monclient: _check_auth_tickets 2024-03-29T11:21:47.369 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:47.391+0000 7f5d5e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:21:47.369 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:47.391+0000 7f5d6fe05640 5 monclient: authenticate success, global_id 4285 2024-03-29T11:21:47.369 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:47.391+0000 7f5d6fe05640 10 monclient: _renew_subs 2024-03-29T11:21:47.369 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:47.391+0000 7f5d6fe05640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-29T11:21:47.369 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:47.391+0000 7f5d6fe05640 1 -- 172.21.15.84:0/3946127605 --> v1:172.21.15.179:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5d6806ff20 con 0x7f5d680a9930 2024-03-29T11:21:47.369 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:47.391+0000 7f5d6fe05640 10 client.4285.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:21:47.370 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:47.391+0000 7f5d6fe05640 10 monclient: _renew_subs 2024-03-29T11:21:47.370 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:47.391+0000 7f5d6fe05640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-29T11:21:47.370 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:47.391+0000 7f5d6fe05640 1 -- 172.21.15.84:0/3946127605 --> v1:172.21.15.179:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f5d681e7620 con 0x7f5d680a9930 2024-03-29T11:21:47.370 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:47.391+0000 7f5d6fe05640 1 librados: init done 2024-03-29T11:21:47.372 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:47.393+0000 7f5d5e7fc640 1 -- 172.21.15.84:0/3946127605 <== mon.1 v1:172.21.15.179:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5d60003cc0 con 0x7f5d680a9930 2024-03-29T11:21:47.372 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:47.393+0000 7f5d5e7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:21:47.372 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:47.393+0000 7f5d5e7fc640 1 -- 172.21.15.84:0/3946127605 <== mon.1 v1:172.21.15.179:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4236868694 0 0) 0x7f5d600040d0 con 0x7f5d680a9930 2024-03-29T11:21:47.372 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:47.393+0000 7f5d5e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:47.372 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:47.393+0000 7f5d5e7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:21:47.372 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:47.393+0000 7f5d5e7fc640 10 monclient: dump: 2024-03-29T11:21:47.372 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T11:21:47.372 INFO:teuthology.misc.health.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:21:47.372 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T11:18:53.864025+0000 2024-03-29T11:21:47.372 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T11:18:53.864025+0000 2024-03-29T11:21:47.372 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:47.372 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:21:47.372 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T11:21:47.373 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T11:21:47.373 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T11:21:47.373 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:21:47.373 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:47.393+0000 7f5d5e7fc640 1 -- 172.21.15.84:0/3946127605 <== mon.1 v1:172.21.15.179:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4033942909 0 0) 0x7f5d600042f0 con 0x7f5d680a9930 2024-03-29T11:21:47.373 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:47.393+0000 7f5d5ffff640 4 set_mon_vals no callback set 2024-03-29T11:21:47.373 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:47.393+0000 7f5d5e7fc640 1 -- 172.21.15.84:0/3946127605 <== mon.1 v1:172.21.15.179:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2808139587 0 0) 0x7f5d60085020 con 0x7f5d680a9930 2024-03-29T11:21:47.373 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:47.393+0000 7f5d5e7fc640 10 client.4285.objecter ms_dispatch 0x7f5d68000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:21:47.373 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:47.393+0000 7f5d5e7fc640 3 client.4285.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:21:47.373 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:47.393+0000 7f5d5e7fc640 3 client.4285.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:21:47.373 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:47.393+0000 7f5d5e7fc640 20 client.4285.objecter dump_active .. 0 homeless 2024-03-29T11:21:47.373 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:47.395+0000 7f5d6d3a0640 10 monclient: get_auth_request con 0x7f5d4c06dd00 auth_method 0 2024-03-29T11:21:47.373 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:47.395+0000 7f5d53fff640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:21:47.373 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:47.395+0000 7f5d53fff640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:21:47.373 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:47.395+0000 7f5d53fff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-29T11:21:47.373 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:47.395+0000 7f5d53fff640 1 -- 172.21.15.84:0/3946127605 --> v1:172.21.15.179:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5d48005080 con 0x7f5d680a9930 2024-03-29T11:21:47.374 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:47.395+0000 7f5d5e7fc640 10 client.4285.objecter ms_handle_connect 0x7f5d4c06dd00 2024-03-29T11:21:47.379 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:47.402+0000 7f5d5e7fc640 1 -- 172.21.15.84:0/3946127605 <== mon.1 v1:172.21.15.179:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f5d60050900 con 0x7f5d680a9930 2024-03-29T11:21:47.379 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:47.402+0000 7f5d5e7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:21:47.379 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:47.402+0000 7f5d5e7fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:21:47.722 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:47.746+0000 7f5d53fff640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:21:47.723 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:47.746+0000 7f5d53fff640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:21:47.723 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:47.746+0000 7f5d53fff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-29T11:21:47.723 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:47.746+0000 7f5d53fff640 1 -- 172.21.15.84:0/3946127605 --> v1:172.21.15.179:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5d48005330 con 0x7f5d680a9930 2024-03-29T11:21:47.723 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:47.747+0000 7f5d5e7fc640 1 -- 172.21.15.84:0/3946127605 <== mon.1 v1:172.21.15.179:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4209477549) 0x7f5d60052830 con 0x7f5d680a9930 2024-03-29T11:21:47.723 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:47.747+0000 7f5d5e7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:21:47.723 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:47.747+0000 7f5d5e7fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:21:47.724 INFO:teuthology.misc.health.smithi084.stdout: 2024-03-29T11:21:47.724 INFO:teuthology.misc.health.smithi084.stdout:{"epoch":10,"fsid":"2c3e002f-9eb0-4860-853b-4ef1ef562213","created":"2024-03-29T11:19:32.029169+0000","modified":"2024-03-29T11:19:34.200299+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:34.200299+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"5c39e276-2034-4716-bbc0-57a634eefa92","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"cd580e99-109f-4bf1-88c0-a5987bc8b9ed","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"11e60a5a-051d-453f-a190-66387cb35592","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"2c329fbe-bbc4-4030-bf87-d711cad04679","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f4213d4a-bd36-4433-91ae-d7e2b02481d3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"f0cf44ea-6055-48af-805b-aed858102ff9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"d895eaa0-cc0b-4ee4-9b5a-2dfcd89a60b7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"9bc5a7bc-8f52-4db9-96d6-de5e68d52eaa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:21:47.727 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:47.750+0000 7f5d53fff640 10 librados: watch_flush enter 2024-03-29T11:21:47.727 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:47.751+0000 7f5d53fff640 10 librados: watch_flush exit 2024-03-29T11:21:47.727 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:47.751+0000 7f5d53fff640 20 client.4285.objecter shutdown clearing up homeless session... 2024-03-29T11:21:47.727 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:47.751+0000 7f5d53fff640 10 client.4285.objecter successfully canceled tick 2024-03-29T11:21:47.728 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:47.751+0000 7f5d53fff640 1 -- 172.21.15.84:0/3946127605 >> v1:172.21.15.179:6816/2981134980 conn(0x7f5d4c06dd00 legacy=0x7f5d4c0701c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:47.728 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:47.751+0000 7f5d53fff640 10 monclient: shutdown 2024-03-29T11:21:47.728 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:47.751+0000 7f5d53fff640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:21:47.728 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:47.751+0000 7f5d53fff640 1 -- 172.21.15.84:0/3946127605 >> v1:172.21.15.179:6789/0 conn(0x7f5d680a9930 legacy=0x7f5d680718c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:47.728 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:47.751+0000 7f5d53fff640 1 -- 172.21.15.84:0/3946127605 shutdown_connections 2024-03-29T11:21:47.728 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:47.751+0000 7f5d53fff640 1 -- 172.21.15.84:0/3946127605 >> 172.21.15.84:0/3946127605 conn(0x7f5d680b35e0 msgr2=0x7f5d68140a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:21:47.728 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:47.751+0000 7f5d53fff640 1 -- 172.21.15.84:0/3946127605 shutdown_connections 2024-03-29T11:21:47.728 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:47.751+0000 7f5d53fff640 1 -- 172.21.15.84:0/3946127605 wait complete. 2024-03-29T11:21:47.729 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:47.752+0000 7f5d53fff640 1 librados: shutdown 2024-03-29T11:21:47.742 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:21:54.550 DEBUG:teuthology.orchestra.run.smithi084:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:21:54.697 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:54.721+0000 7f9a64315640 10 monclient: get_monmap_and_config 2024-03-29T11:21:54.697 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:54.721+0000 7f9a64315640 10 monclient: build_initial_monmap 2024-03-29T11:21:54.698 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:54.721+0000 7f9a64315640 10 monclient: monmap: 2024-03-29T11:21:54.698 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-29T11:21:54.698 INFO:teuthology.misc.health.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:21:54.698 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T11:21:54.722345+0000 2024-03-29T11:21:54.698 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T11:21:54.722345+0000 2024-03-29T11:21:54.698 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-29T11:21:54.698 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:21:54.698 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-29T11:21:54.698 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.noname-c 2024-03-29T11:21:54.698 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-b 2024-03-29T11:21:54.698 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:21:54.699 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:54.722+0000 7f9a64315640 1 Processor -- start 2024-03-29T11:21:54.699 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:54.722+0000 7f9a64315640 1 -- start start 2024-03-29T11:21:54.699 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:54.722+0000 7f9a64315640 10 monclient: init 2024-03-29T11:21:54.699 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:54.723+0000 7f9a64315640 10 monclient: _reopen_session rank -1 2024-03-29T11:21:54.699 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:54.723+0000 7f9a64315640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:21:54.699 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:54.723+0000 7f9a64315640 10 monclient(hunting): picked mon.noname-b con 0x7f9a5c1422b0 addr v1:172.21.15.84:6790/0 2024-03-29T11:21:54.700 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:54.723+0000 7f9a64315640 10 monclient(hunting): picked mon.noname-a con 0x7f9a5c142690 addr v1:172.21.15.84:6789/0 2024-03-29T11:21:54.700 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:54.723+0000 7f9a64315640 10 monclient(hunting): picked mon.noname-c con 0x7f9a5c0b17d0 addr v1:172.21.15.179:6789/0 2024-03-29T11:21:54.700 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:54.723+0000 7f9a64315640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9a5c142c80 con 0x7f9a5c142690 2024-03-29T11:21:54.700 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:54.723+0000 7f9a64315640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9a5c0aae40 con 0x7f9a5c0b17d0 2024-03-29T11:21:54.700 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:54.723+0000 7f9a64315640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9a5c13fbd0 con 0x7f9a5c1422b0 2024-03-29T11:21:54.700 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:54.723+0000 7f9a64315640 10 monclient(hunting): _renew_subs 2024-03-29T11:21:54.700 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:54.723+0000 7f9a64315640 10 monclient(hunting): authenticate will time out at 976.017682s 2024-03-29T11:21:54.700 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:54.723+0000 7f9a618b0640 1 --1- >> v1:172.21.15.84:6789/0 conn(0x7f9a5c142690 0x7f9a5c0b2590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.84:6789/0 says I am v1:172.21.15.84:58336/0 (socket says 172.21.15.84:58336) 2024-03-29T11:21:54.700 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:54.723+0000 7f9a618b0640 1 -- 172.21.15.84:0/805563000 learned_addr learned my addr 172.21.15.84:0/805563000 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-29T11:21:54.701 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:54.724+0000 7f9a610af640 1 -- 172.21.15.84:0/805563000 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1417990244 0 0) 0x7f9a5c13fbd0 con 0x7f9a5c1422b0 2024-03-29T11:21:54.702 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:54.724+0000 7f9a610af640 10 monclient(hunting): _negotiate 2024-03-29T11:21:54.702 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:54.724+0000 7f9a610af640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:21:54.702 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:54.724+0000 7f9a610af640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:21:54.702 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:54.724+0000 7f9a610af640 10 monclient(hunting): my global_id is 4316 2024-03-29T11:21:54.702 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:54.724+0000 7f9a610af640 1 -- 172.21.15.84:0/805563000 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9a48003520 con 0x7f9a5c1422b0 2024-03-29T11:21:54.702 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:54.724+0000 7f9a610af640 1 -- 172.21.15.84:0/805563000 <== mon.1 v1:172.21.15.179:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2896794598 0 0) 0x7f9a5c0aae40 con 0x7f9a5c0b17d0 2024-03-29T11:21:54.702 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:54.724+0000 7f9a610af640 10 monclient(hunting): _negotiate 2024-03-29T11:21:54.702 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:54.724+0000 7f9a610af640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:21:54.702 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:54.724+0000 7f9a610af640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:21:54.702 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:54.724+0000 7f9a610af640 10 monclient(hunting): my global_id is 4288 2024-03-29T11:21:54.702 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:54.724+0000 7f9a610af640 1 -- 172.21.15.84:0/805563000 --> v1:172.21.15.179:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9a5c13fbd0 con 0x7f9a5c0b17d0 2024-03-29T11:21:54.702 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:54.724+0000 7f9a610af640 1 -- 172.21.15.84:0/805563000 <== mon.1 v1:172.21.15.179:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2903427018 0 0) 0x7f9a5c13fbd0 con 0x7f9a5c0b17d0 2024-03-29T11:21:54.703 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:54.724+0000 7f9a610af640 1 -- 172.21.15.84:0/805563000 >> v1:172.21.15.84:6790/0 conn(0x7f9a5c1422b0 legacy=0x7f9a5c13f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:54.703 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:54.724+0000 7f9a610af640 1 -- 172.21.15.84:0/805563000 >> v1:172.21.15.84:6789/0 conn(0x7f9a5c142690 legacy=0x7f9a5c0b2590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:54.703 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:54.724+0000 7f9a610af640 10 monclient: _finish_hunting 0 2024-03-29T11:21:54.703 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:54.724+0000 7f9a610af640 1 monclient: found mon.noname-c 2024-03-29T11:21:54.703 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:54.724+0000 7f9a610af640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:21:54.703 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:54.724+0000 7f9a610af640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.179:6789/0 2024-03-29T11:21:54.703 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:54.724+0000 7f9a610af640 1 -- 172.21.15.84:0/805563000 --> v1:172.21.15.179:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a5c151940 con 0x7f9a5c0b17d0 2024-03-29T11:21:54.703 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:54.724+0000 7f9a610af640 10 monclient: _finish_auth 0 2024-03-29T11:21:54.703 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:54.724+0000 7f9a610af640 10 monclient: _check_auth_tickets 2024-03-29T11:21:54.703 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:54.724+0000 7f9a610af640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:21:54.703 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:54.724+0000 7f9a610af640 1 -- 172.21.15.84:0/805563000 <== mon.1 v1:172.21.15.179:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4236868694 0 0) 0x7f9a580015f0 con 0x7f9a5c0b17d0 2024-03-29T11:21:54.703 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:54.724+0000 7f9a610af640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:54.703 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:54.724+0000 7f9a610af640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:21:54.703 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:54.724+0000 7f9a610af640 10 monclient: dump: 2024-03-29T11:21:54.703 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T11:21:54.703 INFO:teuthology.misc.health.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:21:54.704 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T11:18:53.864025+0000 2024-03-29T11:21:54.704 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T11:18:53.864025+0000 2024-03-29T11:21:54.704 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:54.704 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:21:54.704 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T11:21:54.704 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T11:21:54.704 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T11:21:54.704 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:21:54.704 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:54.724+0000 7f9a64315640 5 monclient: authenticate success, global_id 4288 2024-03-29T11:21:54.704 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:54.724+0000 7f9a64315640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:21:54.704 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:54.725+0000 7f9a610af640 1 -- 172.21.15.84:0/805563000 <== mon.1 v1:172.21.15.179:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9a58001750 con 0x7f9a5c0b17d0 2024-03-29T11:21:54.704 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:54.725+0000 7f9a610af640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:21:54.704 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:54.725+0000 7f9a64315640 10 monclient: get_monmap_and_config success 2024-03-29T11:21:54.704 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:54.725+0000 7f9a64315640 4 set_mon_vals no callback set 2024-03-29T11:21:54.704 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:54.725+0000 7f9a610af640 1 -- 172.21.15.84:0/805563000 <== mon.1 v1:172.21.15.179:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4236868694 0 0) 0x7f9a58001b20 con 0x7f9a5c0b17d0 2024-03-29T11:21:54.705 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:54.725+0000 7f9a610af640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:54.705 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:54.725+0000 7f9a610af640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:21:54.705 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:54.725+0000 7f9a610af640 10 monclient: dump: 2024-03-29T11:21:54.705 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T11:21:54.705 INFO:teuthology.misc.health.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:21:54.705 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T11:18:53.864025+0000 2024-03-29T11:21:54.705 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T11:18:53.864025+0000 2024-03-29T11:21:54.705 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:54.705 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:21:54.705 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T11:21:54.705 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T11:21:54.705 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T11:21:54.705 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:21:54.713 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:54.726+0000 7f9a64315640 10 monclient: shutdown 2024-03-29T11:21:54.713 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:54.726+0000 7f9a64315640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:21:54.713 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:54.726+0000 7f9a64315640 1 -- 172.21.15.84:0/805563000 >> v1:172.21.15.179:6789/0 conn(0x7f9a5c0b17d0 legacy=0x7f9a5c0b2ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:54.713 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:54.726+0000 7f9a64315640 1 -- 172.21.15.84:0/805563000 shutdown_connections 2024-03-29T11:21:54.713 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:54.726+0000 7f9a64315640 1 -- 172.21.15.84:0/805563000 >> 172.21.15.84:0/805563000 conn(0x7f9a5c066a00 msgr2=0x7f9a5c066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:21:54.713 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:54.727+0000 7f9a64315640 1 -- 172.21.15.84:0/805563000 shutdown_connections 2024-03-29T11:21:54.713 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:54.727+0000 7f9a64315640 1 -- 172.21.15.84:0/805563000 wait complete. 2024-03-29T11:21:54.713 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:54.728+0000 7f9a64315640 10 monclient: build_initial_monmap 2024-03-29T11:21:54.714 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:54.728+0000 7f9a64315640 10 monclient: monmap: 2024-03-29T11:21:54.714 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-29T11:21:54.714 INFO:teuthology.misc.health.smithi084.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:21:54.714 INFO:teuthology.misc.health.smithi084.stderr:last_changed 0.000000 2024-03-29T11:21:54.714 INFO:teuthology.misc.health.smithi084.stderr:created 0.000000 2024-03-29T11:21:54.714 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-29T11:21:54.714 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:21:54.714 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-29T11:21:54.714 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.noname-b 2024-03-29T11:21:54.714 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-c 2024-03-29T11:21:54.714 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:21:54.714 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:54.728+0000 7f9a64315640 1 librados: starting msgr at 2024-03-29T11:21:54.714 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:54.728+0000 7f9a64315640 1 librados: starting objecter 2024-03-29T11:21:54.714 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:54.728+0000 7f9a64315640 1 Processor -- start 2024-03-29T11:21:54.714 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:54.728+0000 7f9a64315640 1 -- start start 2024-03-29T11:21:54.715 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:54.728+0000 7f9a64315640 1 librados: setting wanted keys 2024-03-29T11:21:54.715 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:54.728+0000 7f9a64315640 1 librados: calling monclient init 2024-03-29T11:21:54.715 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:54.728+0000 7f9a64315640 10 monclient: init 2024-03-29T11:21:54.715 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:54.729+0000 7f9a64315640 10 monclient: _reopen_session rank -1 2024-03-29T11:21:54.715 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:54.729+0000 7f9a64315640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:21:54.715 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:54.729+0000 7f9a64315640 10 monclient(hunting): picked mon.noname-c con 0x7f9a5c0b17d0 addr v1:172.21.15.84:6790/0 2024-03-29T11:21:54.715 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:54.729+0000 7f9a64315640 10 monclient(hunting): picked mon.noname-b con 0x7f9a5c1422b0 addr v1:172.21.15.179:6789/0 2024-03-29T11:21:54.715 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:54.729+0000 7f9a64315640 10 monclient(hunting): picked mon.noname-a con 0x7f9a5c142690 addr v1:172.21.15.84:6789/0 2024-03-29T11:21:54.715 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:54.729+0000 7f9a64315640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9a5c12f560 con 0x7f9a5c142690 2024-03-29T11:21:54.715 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:54.730+0000 7f9a64315640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9a5c06ee40 con 0x7f9a5c1422b0 2024-03-29T11:21:54.715 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:54.730+0000 7f9a64315640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9a5c06f120 con 0x7f9a5c0b17d0 2024-03-29T11:21:54.715 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:54.730+0000 7f9a64315640 10 monclient(hunting): _renew_subs 2024-03-29T11:21:54.715 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:54.730+0000 7f9a64315640 10 monclient(hunting): authenticate will time out at 976.024329s 2024-03-29T11:21:54.715 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:54.730+0000 7f9a618b0640 1 --1- >> v1:172.21.15.179:6789/0 conn(0x7f9a5c1422b0 0x7f9a5c12e740 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.179:6789/0 says I am v1:172.21.15.84:49430/0 (socket says 172.21.15.84:49430) 2024-03-29T11:21:54.715 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:54.730+0000 7f9a618b0640 1 -- 172.21.15.84:0/4024728604 learned_addr learned my addr 172.21.15.84:0/4024728604 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-29T11:21:54.716 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:54.730+0000 7f9a52ffd640 10 client.?.objecter ms_handle_connect 0x7f9a5c1422b0 2024-03-29T11:21:54.716 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:54.730+0000 7f9a52ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:21:54.716 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:54.730+0000 7f9a52ffd640 10 client.?.objecter ms_handle_connect 0x7f9a5c0b17d0 2024-03-29T11:21:54.716 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:54.730+0000 7f9a52ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:21:54.716 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:54.730+0000 7f9a52ffd640 1 -- 172.21.15.84:0/4024728604 <== mon.1 v1:172.21.15.179:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1892533760 0 0) 0x7f9a5c06ee40 con 0x7f9a5c1422b0 2024-03-29T11:21:54.716 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:54.731+0000 7f9a52ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:21:54.716 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:54.731+0000 7f9a52ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:21:54.716 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:54.731+0000 7f9a52ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:21:54.716 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:54.731+0000 7f9a52ffd640 10 monclient(hunting): my global_id is 4291 2024-03-29T11:21:54.716 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:54.731+0000 7f9a52ffd640 1 -- 172.21.15.84:0/4024728604 --> v1:172.21.15.179:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9a40004eb0 con 0x7f9a5c1422b0 2024-03-29T11:21:54.716 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:54.731+0000 7f9a52ffd640 10 client.?.objecter ms_handle_connect 0x7f9a5c142690 2024-03-29T11:21:54.716 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:54.731+0000 7f9a52ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:21:54.716 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:54.731+0000 7f9a52ffd640 1 -- 172.21.15.84:0/4024728604 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2092295939 0 0) 0x7f9a5c06f120 con 0x7f9a5c0b17d0 2024-03-29T11:21:54.716 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:54.731+0000 7f9a52ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:21:54.716 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:54.731+0000 7f9a52ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:21:54.716 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:54.731+0000 7f9a52ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:21:54.717 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:54.731+0000 7f9a52ffd640 10 monclient(hunting): my global_id is 4319 2024-03-29T11:21:54.717 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:54.731+0000 7f9a52ffd640 1 -- 172.21.15.84:0/4024728604 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9a5c06ee40 con 0x7f9a5c0b17d0 2024-03-29T11:21:54.717 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:54.731+0000 7f9a52ffd640 1 -- 172.21.15.84:0/4024728604 <== mon.0 v1:172.21.15.84:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 738883805 0 0) 0x7f9a5c12f560 con 0x7f9a5c142690 2024-03-29T11:21:54.717 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:54.731+0000 7f9a52ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:21:54.717 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:54.731+0000 7f9a52ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:21:54.717 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:54.731+0000 7f9a52ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:21:54.717 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:54.732+0000 7f9a52ffd640 10 monclient(hunting): my global_id is 4326 2024-03-29T11:21:54.717 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:54.732+0000 7f9a52ffd640 1 -- 172.21.15.84:0/4024728604 --> v1:172.21.15.84:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9a5c06f120 con 0x7f9a5c142690 2024-03-29T11:21:54.717 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:54.732+0000 7f9a52ffd640 1 -- 172.21.15.84:0/4024728604 <== mon.1 v1:172.21.15.179:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2395263597 0 0) 0x7f9a40004eb0 con 0x7f9a5c1422b0 2024-03-29T11:21:54.717 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:54.732+0000 7f9a52ffd640 1 -- 172.21.15.84:0/4024728604 --> v1:172.21.15.179:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9a5c12f560 con 0x7f9a5c1422b0 2024-03-29T11:21:54.717 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:54.732+0000 7f9a52ffd640 1 -- 172.21.15.84:0/4024728604 <== mon.1 v1:172.21.15.179:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4236868694 0 0) 0x7f9a4c001810 con 0x7f9a5c1422b0 2024-03-29T11:21:54.717 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:54.732+0000 7f9a52ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:54.717 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:54.732+0000 7f9a52ffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:21:54.717 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:54.732+0000 7f9a52ffd640 10 monclient(hunting): dump: 2024-03-29T11:21:54.717 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T11:21:54.718 INFO:teuthology.misc.health.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:21:54.718 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T11:18:53.864025+0000 2024-03-29T11:21:54.718 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T11:18:53.864025+0000 2024-03-29T11:21:54.718 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:54.718 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:21:54.718 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T11:21:54.718 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T11:21:54.718 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T11:21:54.718 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:21:54.718 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:54.732+0000 7f9a52ffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:21:54.718 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:54.732+0000 7f9a52ffd640 1 -- 172.21.15.84:0/4024728604 <== mon.2 v1:172.21.15.84:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2482876299 0 0) 0x7f9a5c06ee40 con 0x7f9a5c0b17d0 2024-03-29T11:21:54.718 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:54.732+0000 7f9a52ffd640 1 -- 172.21.15.84:0/4024728604 --> v1:172.21.15.84:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9a40004eb0 con 0x7f9a5c0b17d0 2024-03-29T11:21:54.718 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:54.732+0000 7f9a52ffd640 1 -- 172.21.15.84:0/4024728604 <== mon.2 v1:172.21.15.84:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4236868694 0 0) 0x7f9a54001c50 con 0x7f9a5c0b17d0 2024-03-29T11:21:54.718 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:54.732+0000 7f9a52ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:54.719 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:54.732+0000 7f9a52ffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:21:54.719 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:54.732+0000 7f9a52ffd640 10 monclient(hunting): dump: 2024-03-29T11:21:54.719 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T11:21:54.719 INFO:teuthology.misc.health.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:21:54.719 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T11:18:53.864025+0000 2024-03-29T11:21:54.719 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T11:18:53.864025+0000 2024-03-29T11:21:54.719 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:54.719 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:21:54.719 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T11:21:54.719 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T11:21:54.719 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T11:21:54.719 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:21:54.720 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:54.732+0000 7f9a52ffd640 1 -- 172.21.15.84:0/4024728604 <== mon.1 v1:172.21.15.179:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2210003661 0 0) 0x7f9a5c12f560 con 0x7f9a5c1422b0 2024-03-29T11:21:54.720 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:54.733+0000 7f9a52ffd640 1 -- 172.21.15.84:0/4024728604 >> v1:172.21.15.84:6790/0 conn(0x7f9a5c0b17d0 legacy=0x7f9a5c132190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:54.720 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:54.733+0000 7f9a52ffd640 1 -- 172.21.15.84:0/4024728604 >> v1:172.21.15.84:6789/0 conn(0x7f9a5c142690 legacy=0x7f9a5c12ee50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:54.720 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:54.733+0000 7f9a52ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:21:54.720 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:54.733+0000 7f9a52ffd640 1 monclient: found mon.b 2024-03-29T11:21:54.720 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:54.733+0000 7f9a52ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:21:54.720 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:54.733+0000 7f9a52ffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-29T11:21:54.720 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:54.733+0000 7f9a52ffd640 1 -- 172.21.15.84:0/4024728604 --> v1:172.21.15.179:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a5c131870 con 0x7f9a5c1422b0 2024-03-29T11:21:54.720 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:54.733+0000 7f9a52ffd640 10 monclient: _finish_auth 0 2024-03-29T11:21:54.720 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:54.733+0000 7f9a52ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:21:54.720 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:54.733+0000 7f9a52ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:21:54.720 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:54.733+0000 7f9a64315640 5 monclient: authenticate success, global_id 4291 2024-03-29T11:21:54.720 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:54.733+0000 7f9a64315640 10 monclient: _renew_subs 2024-03-29T11:21:54.720 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:54.733+0000 7f9a64315640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-29T11:21:54.720 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:54.733+0000 7f9a64315640 1 -- 172.21.15.84:0/4024728604 --> v1:172.21.15.179:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9a5c1328a0 con 0x7f9a5c1422b0 2024-03-29T11:21:54.720 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:54.733+0000 7f9a64315640 10 client.4291.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:21:54.720 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:54.733+0000 7f9a64315640 10 monclient: _renew_subs 2024-03-29T11:21:54.721 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:54.733+0000 7f9a64315640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-29T11:21:54.721 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:54.733+0000 7f9a64315640 1 -- 172.21.15.84:0/4024728604 --> v1:172.21.15.179:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f9a5c06e070 con 0x7f9a5c1422b0 2024-03-29T11:21:54.721 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:54.733+0000 7f9a64315640 1 librados: init done 2024-03-29T11:21:54.721 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:54.734+0000 7f9a52ffd640 1 -- 172.21.15.84:0/4024728604 <== mon.1 v1:172.21.15.179:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9a4c001d00 con 0x7f9a5c1422b0 2024-03-29T11:21:54.721 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:54.734+0000 7f9a52ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:21:54.721 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:54.734+0000 7f9a608ae640 4 set_mon_vals no callback set 2024-03-29T11:21:54.721 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:54.735+0000 7f9a64315640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:21:54.721 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:54.735+0000 7f9a52ffd640 1 -- 172.21.15.84:0/4024728604 <== mon.1 v1:172.21.15.179:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4236868694 0 0) 0x7f9a4c004980 con 0x7f9a5c1422b0 2024-03-29T11:21:54.721 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:54.735+0000 7f9a64315640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:21:54.721 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:54.735+0000 7f9a64315640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-29T11:21:54.721 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:54.735+0000 7f9a64315640 1 -- 172.21.15.84:0/4024728604 --> v1:172.21.15.179:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9a44005080 con 0x7f9a5c1422b0 2024-03-29T11:21:54.721 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:54.735+0000 7f9a52ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:54.721 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:54.735+0000 7f9a52ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:21:54.721 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:54.735+0000 7f9a52ffd640 10 monclient: dump: 2024-03-29T11:21:54.721 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T11:21:54.722 INFO:teuthology.misc.health.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:21:54.722 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T11:18:53.864025+0000 2024-03-29T11:21:54.722 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T11:18:53.864025+0000 2024-03-29T11:21:54.722 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:54.722 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:21:54.722 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T11:21:54.722 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T11:21:54.722 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T11:21:54.722 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:21:54.722 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:54.736+0000 7f9a52ffd640 1 -- 172.21.15.84:0/4024728604 <== mon.1 v1:172.21.15.179:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4033942909 0 0) 0x7f9a4c004ba0 con 0x7f9a5c1422b0 2024-03-29T11:21:54.725 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:54.737+0000 7f9a52ffd640 1 -- 172.21.15.84:0/4024728604 <== mon.1 v1:172.21.15.179:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2808139587 0 0) 0x7f9a4c084a10 con 0x7f9a5c1422b0 2024-03-29T11:21:54.725 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:54.737+0000 7f9a52ffd640 10 client.4291.objecter ms_dispatch 0x7f9a5c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:21:54.725 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:54.737+0000 7f9a52ffd640 3 client.4291.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:21:54.725 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:54.737+0000 7f9a52ffd640 3 client.4291.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:21:54.725 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:54.737+0000 7f9a52ffd640 20 client.4291.objecter dump_active .. 0 homeless 2024-03-29T11:21:54.726 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:54.738+0000 7f9a620b1640 10 monclient: get_auth_request con 0x7f9a4006dce0 auth_method 0 2024-03-29T11:21:54.726 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:54.738+0000 7f9a52ffd640 10 client.4291.objecter ms_handle_connect 0x7f9a4006dce0 2024-03-29T11:21:54.726 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:54.745+0000 7f9a52ffd640 1 -- 172.21.15.84:0/4024728604 <== mon.1 v1:172.21.15.179:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f9a4c0502f0 con 0x7f9a5c1422b0 2024-03-29T11:21:54.726 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:54.745+0000 7f9a52ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:21:54.727 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:54.745+0000 7f9a52ffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:21:55.073 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:55.096+0000 7f9a64315640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:21:55.073 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:55.096+0000 7f9a64315640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:21:55.073 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:55.096+0000 7f9a64315640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-29T11:21:55.073 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:55.096+0000 7f9a64315640 1 -- 172.21.15.84:0/4024728604 --> v1:172.21.15.179:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9a44005330 con 0x7f9a5c1422b0 2024-03-29T11:21:55.074 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:55.098+0000 7f9a52ffd640 1 -- 172.21.15.84:0/4024728604 <== mon.1 v1:172.21.15.179:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4209477549) 0x7f9a4c052220 con 0x7f9a5c1422b0 2024-03-29T11:21:55.074 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:55.098+0000 7f9a52ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:21:55.074 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:55.098+0000 7f9a52ffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:21:55.075 INFO:teuthology.misc.health.smithi084.stdout: 2024-03-29T11:21:55.075 INFO:teuthology.misc.health.smithi084.stdout:{"epoch":10,"fsid":"2c3e002f-9eb0-4860-853b-4ef1ef562213","created":"2024-03-29T11:19:32.029169+0000","modified":"2024-03-29T11:19:34.200299+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:34.200299+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"5c39e276-2034-4716-bbc0-57a634eefa92","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"cd580e99-109f-4bf1-88c0-a5987bc8b9ed","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"11e60a5a-051d-453f-a190-66387cb35592","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"2c329fbe-bbc4-4030-bf87-d711cad04679","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f4213d4a-bd36-4433-91ae-d7e2b02481d3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"f0cf44ea-6055-48af-805b-aed858102ff9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"d895eaa0-cc0b-4ee4-9b5a-2dfcd89a60b7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"9bc5a7bc-8f52-4db9-96d6-de5e68d52eaa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:21:55.077 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:55.101+0000 7f9a64315640 10 librados: watch_flush enter 2024-03-29T11:21:55.077 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:55.101+0000 7f9a64315640 10 librados: watch_flush exit 2024-03-29T11:21:55.077 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:55.101+0000 7f9a64315640 20 client.4291.objecter shutdown clearing up homeless session... 2024-03-29T11:21:55.077 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:55.101+0000 7f9a64315640 10 client.4291.objecter successfully canceled tick 2024-03-29T11:21:55.078 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:55.101+0000 7f9a64315640 1 -- 172.21.15.84:0/4024728604 >> v1:172.21.15.179:6816/2981134980 conn(0x7f9a4006dce0 legacy=0x7f9a400701a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:55.078 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:55.101+0000 7f9a64315640 10 monclient: shutdown 2024-03-29T11:21:55.078 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:55.101+0000 7f9a64315640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:21:55.078 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:55.101+0000 7f9a64315640 1 -- 172.21.15.84:0/4024728604 >> v1:172.21.15.179:6789/0 conn(0x7f9a5c1422b0 legacy=0x7f9a5c12e740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:55.078 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:55.101+0000 7f9a64315640 1 -- 172.21.15.84:0/4024728604 shutdown_connections 2024-03-29T11:21:55.079 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:55.101+0000 7f9a64315640 1 -- 172.21.15.84:0/4024728604 >> 172.21.15.84:0/4024728604 conn(0x7f9a5c066a00 msgr2=0x7f9a5c0b2490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:21:55.079 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:55.101+0000 7f9a64315640 1 -- 172.21.15.84:0/4024728604 shutdown_connections 2024-03-29T11:21:55.079 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:55.102+0000 7f9a64315640 1 -- 172.21.15.84:0/4024728604 wait complete. 2024-03-29T11:21:55.079 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:21:55.102+0000 7f9a64315640 1 librados: shutdown 2024-03-29T11:21:55.092 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:22:01.898 DEBUG:teuthology.orchestra.run.smithi084:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:22:02.046 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:02.070+0000 7fedd9e4a640 10 monclient: get_monmap_and_config 2024-03-29T11:22:02.046 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:02.070+0000 7fedd9e4a640 10 monclient: build_initial_monmap 2024-03-29T11:22:02.046 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:02.070+0000 7fedd9e4a640 10 monclient: monmap: 2024-03-29T11:22:02.047 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-29T11:22:02.047 INFO:teuthology.misc.health.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:22:02.047 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T11:22:02.071135+0000 2024-03-29T11:22:02.047 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T11:22:02.071135+0000 2024-03-29T11:22:02.047 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-29T11:22:02.047 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:22:02.047 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-29T11:22:02.047 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.noname-c 2024-03-29T11:22:02.047 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-b 2024-03-29T11:22:02.047 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:22:02.048 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:02.071+0000 7fedd9e4a640 1 Processor -- start 2024-03-29T11:22:02.048 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:02.071+0000 7fedd9e4a640 1 -- start start 2024-03-29T11:22:02.048 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:02.071+0000 7fedd9e4a640 10 monclient: init 2024-03-29T11:22:02.048 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:02.072+0000 7fedd9e4a640 10 monclient: _reopen_session rank -1 2024-03-29T11:22:02.048 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:02.072+0000 7fedd9e4a640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:22:02.048 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:02.072+0000 7fedd9e4a640 10 monclient(hunting): picked mon.noname-a con 0x7fedd40b1f60 addr v1:172.21.15.84:6789/0 2024-03-29T11:22:02.049 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:02.072+0000 7fedd9e4a640 10 monclient(hunting): picked mon.noname-b con 0x7fedd4071eb0 addr v1:172.21.15.84:6790/0 2024-03-29T11:22:02.049 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:02.072+0000 7fedd9e4a640 10 monclient(hunting): picked mon.noname-c con 0x7fedd4143d10 addr v1:172.21.15.179:6789/0 2024-03-29T11:22:02.050 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:02.072+0000 7fedd9e4a640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fedd406ef80 con 0x7fedd40b1f60 2024-03-29T11:22:02.051 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:02.072+0000 7fedd9e4a640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fedd406f260 con 0x7fedd4143d10 2024-03-29T11:22:02.051 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:02.072+0000 7fedd9e4a640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fedd40b40c0 con 0x7fedd4071eb0 2024-03-29T11:22:02.051 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:02.072+0000 7fedd9e4a640 10 monclient(hunting): _renew_subs 2024-03-29T11:22:02.051 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:02.072+0000 7fedd9e4a640 10 monclient(hunting): authenticate will time out at 983.366638s 2024-03-29T11:22:02.052 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:02.072+0000 7fedd3fff640 1 --1- >> v1:172.21.15.179:6789/0 conn(0x7fedd4143d10 0x7fedd4146b80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.179:6789/0 says I am v1:172.21.15.84:59906/0 (socket says 172.21.15.84:59906) 2024-03-29T11:22:02.052 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:02.072+0000 7fedd3fff640 1 -- 172.21.15.84:0/1525911148 learned_addr learned my addr 172.21.15.84:0/1525911148 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-29T11:22:02.052 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:02.074+0000 7fedd27fc640 1 -- 172.21.15.84:0/1525911148 <== mon.1 v1:172.21.15.179:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2259642227 0 0) 0x7fedd406f260 con 0x7fedd4143d10 2024-03-29T11:22:02.052 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:02.074+0000 7fedd27fc640 10 monclient(hunting): _negotiate 2024-03-29T11:22:02.052 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:02.074+0000 7fedd27fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:22:02.052 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:02.074+0000 7fedd27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:22:02.052 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:02.074+0000 7fedd27fc640 10 monclient(hunting): my global_id is 4294 2024-03-29T11:22:02.052 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:02.074+0000 7fedd27fc640 1 -- 172.21.15.84:0/1525911148 --> v1:172.21.15.179:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fedbc003520 con 0x7fedd4143d10 2024-03-29T11:22:02.056 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:02.077+0000 7fedd27fc640 1 -- 172.21.15.84:0/1525911148 <== mon.1 v1:172.21.15.179:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1116818643 0 0) 0x7fedbc003520 con 0x7fedd4143d10 2024-03-29T11:22:02.056 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:02.077+0000 7fedd27fc640 1 -- 172.21.15.84:0/1525911148 >> v1:172.21.15.84:6790/0 conn(0x7fedd4071eb0 legacy=0x7fedd4146470 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:22:02.056 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:02.077+0000 7fedd27fc640 1 -- 172.21.15.84:0/1525911148 >> v1:172.21.15.84:6789/0 conn(0x7fedd40b1f60 legacy=0x7fedd4143600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:22:02.056 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:02.077+0000 7fedd27fc640 10 monclient: _finish_hunting 0 2024-03-29T11:22:02.056 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:02.077+0000 7fedd27fc640 1 monclient: found mon.noname-c 2024-03-29T11:22:02.057 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:02.077+0000 7fedd27fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:22:02.057 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:02.077+0000 7fedd27fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.179:6789/0 2024-03-29T11:22:02.057 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:02.077+0000 7fedd27fc640 1 -- 172.21.15.84:0/1525911148 --> v1:172.21.15.179:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fedd4155d70 con 0x7fedd4143d10 2024-03-29T11:22:02.057 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:02.077+0000 7fedd27fc640 10 monclient: _finish_auth 0 2024-03-29T11:22:02.057 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:02.077+0000 7fedd27fc640 10 monclient: _check_auth_tickets 2024-03-29T11:22:02.057 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:02.077+0000 7fedd27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:22:02.057 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:02.077+0000 7fedd27fc640 1 -- 172.21.15.84:0/1525911148 <== mon.1 v1:172.21.15.179:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4236868694 0 0) 0x7fedc4001690 con 0x7fedd4143d10 2024-03-29T11:22:02.057 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:02.077+0000 7fedd27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:02.057 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:02.077+0000 7fedd27fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:22:02.057 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:02.077+0000 7fedd27fc640 10 monclient: dump: 2024-03-29T11:22:02.057 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T11:22:02.057 INFO:teuthology.misc.health.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:22:02.057 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T11:18:53.864025+0000 2024-03-29T11:22:02.057 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T11:18:53.864025+0000 2024-03-29T11:22:02.057 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:02.058 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:22:02.058 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T11:22:02.058 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T11:22:02.058 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T11:22:02.058 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:22:02.058 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:02.078+0000 7fedd27fc640 1 -- 172.21.15.84:0/1525911148 <== mon.1 v1:172.21.15.179:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fedc4001920 con 0x7fedd4143d10 2024-03-29T11:22:02.058 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:02.078+0000 7fedd27fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:22:02.058 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:02.078+0000 7fedd27fc640 1 -- 172.21.15.84:0/1525911148 <== mon.1 v1:172.21.15.179:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4236868694 0 0) 0x7fedc4001cb0 con 0x7fedd4143d10 2024-03-29T11:22:02.058 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:02.078+0000 7fedd27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:02.058 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:02.078+0000 7fedd27fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:22:02.058 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:02.078+0000 7fedd27fc640 10 monclient: dump: 2024-03-29T11:22:02.058 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T11:22:02.058 INFO:teuthology.misc.health.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:22:02.058 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T11:18:53.864025+0000 2024-03-29T11:22:02.059 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T11:18:53.864025+0000 2024-03-29T11:22:02.059 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:02.059 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:22:02.059 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T11:22:02.059 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T11:22:02.059 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T11:22:02.059 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:22:02.059 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:02.079+0000 7fedd9e4a640 5 monclient: authenticate success, global_id 4294 2024-03-29T11:22:02.059 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:02.079+0000 7fedd9e4a640 10 monclient: get_monmap_and_config success 2024-03-29T11:22:02.059 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:02.079+0000 7fedd9e4a640 4 set_mon_vals no callback set 2024-03-29T11:22:02.059 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:02.079+0000 7fedd9e4a640 10 monclient: shutdown 2024-03-29T11:22:02.059 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:02.079+0000 7fedd9e4a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:22:02.059 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:02.079+0000 7fedd9e4a640 1 -- 172.21.15.84:0/1525911148 >> v1:172.21.15.179:6789/0 conn(0x7fedd4143d10 legacy=0x7fedd4146b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:02.059 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:02.079+0000 7fedd9e4a640 1 -- 172.21.15.84:0/1525911148 shutdown_connections 2024-03-29T11:22:02.059 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:02.080+0000 7fedd9e4a640 1 -- 172.21.15.84:0/1525911148 >> 172.21.15.84:0/1525911148 conn(0x7fedd4066a00 msgr2=0x7fedd4066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:22:02.060 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:02.080+0000 7fedd9e4a640 1 -- 172.21.15.84:0/1525911148 shutdown_connections 2024-03-29T11:22:02.060 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:02.081+0000 7fedd9e4a640 1 -- 172.21.15.84:0/1525911148 wait complete. 2024-03-29T11:22:02.060 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:02.081+0000 7fedd9e4a640 10 monclient: build_initial_monmap 2024-03-29T11:22:02.060 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:02.081+0000 7fedd9e4a640 10 monclient: monmap: 2024-03-29T11:22:02.060 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-29T11:22:02.061 INFO:teuthology.misc.health.smithi084.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:22:02.061 INFO:teuthology.misc.health.smithi084.stderr:last_changed 0.000000 2024-03-29T11:22:02.061 INFO:teuthology.misc.health.smithi084.stderr:created 0.000000 2024-03-29T11:22:02.061 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-29T11:22:02.061 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:22:02.061 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-29T11:22:02.061 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.noname-b 2024-03-29T11:22:02.061 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-c 2024-03-29T11:22:02.061 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:22:02.061 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:02.082+0000 7fedd9e4a640 1 librados: starting msgr at 2024-03-29T11:22:02.061 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:02.082+0000 7fedd9e4a640 1 librados: starting objecter 2024-03-29T11:22:02.061 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:02.082+0000 7fedd9e4a640 1 Processor -- start 2024-03-29T11:22:02.061 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:02.082+0000 7fedd9e4a640 1 -- start start 2024-03-29T11:22:02.061 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:02.082+0000 7fedd9e4a640 1 librados: setting wanted keys 2024-03-29T11:22:02.061 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:02.082+0000 7fedd9e4a640 1 librados: calling monclient init 2024-03-29T11:22:02.062 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:02.082+0000 7fedd9e4a640 10 monclient: init 2024-03-29T11:22:02.062 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:02.083+0000 7fedd9e4a640 10 monclient: _reopen_session rank -1 2024-03-29T11:22:02.062 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:02.083+0000 7fedd9e4a640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:22:02.062 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:02.083+0000 7fedd9e4a640 10 monclient(hunting): picked mon.noname-c con 0x7fedd40b1f60 addr v1:172.21.15.84:6790/0 2024-03-29T11:22:02.062 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:02.083+0000 7fedd9e4a640 10 monclient(hunting): picked mon.noname-b con 0x7fedd4143d10 addr v1:172.21.15.179:6789/0 2024-03-29T11:22:02.062 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:02.083+0000 7fedd9e4a640 10 monclient(hunting): picked mon.noname-a con 0x7fedd4136e10 addr v1:172.21.15.84:6789/0 2024-03-29T11:22:02.062 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:02.083+0000 7fedd9e4a640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fedd4070190 con 0x7fedd4136e10 2024-03-29T11:22:02.062 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:02.083+0000 7fedd9e4a640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fedd4071820 con 0x7fedd4143d10 2024-03-29T11:22:02.062 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:02.083+0000 7fedd9e4a640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fedd4070970 con 0x7fedd40b1f60 2024-03-29T11:22:02.062 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:02.083+0000 7fedd9e4a640 10 monclient(hunting): _renew_subs 2024-03-29T11:22:02.062 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:02.083+0000 7fedd9e4a640 10 monclient(hunting): authenticate will time out at 983.377622s 2024-03-29T11:22:02.064 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:02.087+0000 7fedd37fe640 1 --1- >> v1:172.21.15.84:6790/0 conn(0x7fedd40b1f60 0x7fedd408b5d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.84:6790/0 says I am v1:172.21.15.84:36051/0 (socket says 172.21.15.84:36051) 2024-03-29T11:22:02.064 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:02.087+0000 7fedd37fe640 1 -- 172.21.15.84:0/770339044 learned_addr learned my addr 172.21.15.84:0/770339044 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-29T11:22:02.064 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:02.087+0000 7fedbbfff640 10 client.?.objecter ms_handle_connect 0x7fedd40b1f60 2024-03-29T11:22:02.064 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:02.087+0000 7fedbbfff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:22:02.064 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:02.087+0000 7fedbbfff640 1 -- 172.21.15.84:0/770339044 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 573040936 0 0) 0x7fedd4070970 con 0x7fedd40b1f60 2024-03-29T11:22:02.065 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:02.087+0000 7fedbbfff640 10 monclient(hunting): _negotiate 2024-03-29T11:22:02.065 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:02.087+0000 7fedbbfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:22:02.065 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:02.087+0000 7fedbbfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:22:02.065 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:02.087+0000 7fedbbfff640 10 monclient(hunting): my global_id is 4322 2024-03-29T11:22:02.065 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:02.087+0000 7fedbbfff640 1 -- 172.21.15.84:0/770339044 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fedc0004eb0 con 0x7fedd40b1f60 2024-03-29T11:22:02.065 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:02.088+0000 7fedbbfff640 1 -- 172.21.15.84:0/770339044 <== mon.2 v1:172.21.15.84:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4020337774 0 0) 0x7fedc0004eb0 con 0x7fedd40b1f60 2024-03-29T11:22:02.065 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:02.088+0000 7fedbbfff640 1 -- 172.21.15.84:0/770339044 --> v1:172.21.15.84:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fedd4070970 con 0x7fedd40b1f60 2024-03-29T11:22:02.065 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:02.088+0000 7fedbbfff640 1 -- 172.21.15.84:0/770339044 <== mon.2 v1:172.21.15.84:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4236868694 0 0) 0x7fedc8001960 con 0x7fedd40b1f60 2024-03-29T11:22:02.065 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:02.088+0000 7fedbbfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:02.065 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:02.088+0000 7fedbbfff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:22:02.065 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:02.088+0000 7fedbbfff640 10 monclient(hunting): dump: 2024-03-29T11:22:02.065 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T11:22:02.065 INFO:teuthology.misc.health.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:22:02.065 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T11:18:53.864025+0000 2024-03-29T11:22:02.066 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T11:18:53.864025+0000 2024-03-29T11:22:02.066 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:02.066 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:22:02.066 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T11:22:02.066 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T11:22:02.066 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T11:22:02.066 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:22:02.066 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:02.088+0000 7fedbbfff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:22:02.067 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:02.088+0000 7fedbbfff640 10 client.?.objecter ms_handle_connect 0x7fedd4143d10 2024-03-29T11:22:02.067 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:02.088+0000 7fedbbfff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:22:02.067 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:02.088+0000 7fedbbfff640 10 client.?.objecter ms_handle_connect 0x7fedd4136e10 2024-03-29T11:22:02.068 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:02.088+0000 7fedbbfff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:22:02.068 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:02.088+0000 7fedbbfff640 1 -- 172.21.15.84:0/770339044 <== mon.2 v1:172.21.15.84:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3267128627 0 0) 0x7fedd4070970 con 0x7fedd40b1f60 2024-03-29T11:22:02.068 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:02.088+0000 7fedbbfff640 1 -- 172.21.15.84:0/770339044 >> v1:172.21.15.179:6789/0 conn(0x7fedd4143d10 legacy=0x7fedd4153500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:02.068 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:02.088+0000 7fedbbfff640 1 -- 172.21.15.84:0/770339044 >> v1:172.21.15.84:6789/0 conn(0x7fedd4136e10 legacy=0x7fedd412d8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:02.068 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:02.088+0000 7fedbbfff640 10 monclient: _finish_hunting 0 2024-03-29T11:22:02.068 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:02.088+0000 7fedbbfff640 1 monclient: found mon.c 2024-03-29T11:22:02.068 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:02.088+0000 7fedbbfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:22:02.068 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:02.088+0000 7fedbbfff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-29T11:22:02.068 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:02.088+0000 7fedbbfff640 1 -- 172.21.15.84:0/770339044 --> v1:172.21.15.84:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fedd40715a0 con 0x7fedd40b1f60 2024-03-29T11:22:02.068 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:02.088+0000 7fedbbfff640 10 monclient: _finish_auth 0 2024-03-29T11:22:02.068 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:02.088+0000 7fedbbfff640 10 monclient: _check_auth_tickets 2024-03-29T11:22:02.068 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:02.088+0000 7fedbbfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:22:02.068 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:02.089+0000 7fedd9e4a640 5 monclient: authenticate success, global_id 4322 2024-03-29T11:22:02.068 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:02.089+0000 7fedd9e4a640 10 monclient: _renew_subs 2024-03-29T11:22:02.069 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:02.089+0000 7fedd9e4a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-29T11:22:02.069 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:02.089+0000 7fedd9e4a640 1 -- 172.21.15.84:0/770339044 --> v1:172.21.15.84:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fedd4070bc0 con 0x7fedd40b1f60 2024-03-29T11:22:02.069 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:02.089+0000 7fedd9e4a640 10 client.4322.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:22:02.069 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:02.089+0000 7fedd9e4a640 10 monclient: _renew_subs 2024-03-29T11:22:02.069 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:02.089+0000 7fedd9e4a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-29T11:22:02.069 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:02.089+0000 7fedd9e4a640 1 -- 172.21.15.84:0/770339044 --> v1:172.21.15.84:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fedd4137d60 con 0x7fedd40b1f60 2024-03-29T11:22:02.069 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:02.089+0000 7fedd9e4a640 1 librados: init done 2024-03-29T11:22:02.069 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:02.091+0000 7fedd9e4a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:22:02.069 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:02.091+0000 7fedd9e4a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:22:02.069 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:02.091+0000 7fedd9e4a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-29T11:22:02.069 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:02.091+0000 7fedd9e4a640 1 -- 172.21.15.84:0/770339044 --> v1:172.21.15.84:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fedd406fde0 con 0x7fedd40b1f60 2024-03-29T11:22:02.069 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:02.091+0000 7fedbbfff640 1 -- 172.21.15.84:0/770339044 <== mon.2 v1:172.21.15.84:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fedc8001ac0 con 0x7fedd40b1f60 2024-03-29T11:22:02.069 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:02.091+0000 7fedbbfff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:22:02.069 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:02.091+0000 7fedbbfff640 1 -- 172.21.15.84:0/770339044 <== mon.2 v1:172.21.15.84:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4236868694 0 0) 0x7fedc8001d80 con 0x7fedd40b1f60 2024-03-29T11:22:02.070 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:02.091+0000 7fedbbfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:02.070 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:02.091+0000 7fedbbfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:22:02.070 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:02.091+0000 7fedbbfff640 10 monclient: dump: 2024-03-29T11:22:02.070 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T11:22:02.070 INFO:teuthology.misc.health.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:22:02.070 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T11:18:53.864025+0000 2024-03-29T11:22:02.070 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T11:18:53.864025+0000 2024-03-29T11:22:02.070 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:02.070 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:22:02.070 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T11:22:02.070 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T11:22:02.070 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T11:22:02.070 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:22:02.071 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:02.091+0000 7fedbbfff640 1 -- 172.21.15.84:0/770339044 <== mon.2 v1:172.21.15.84:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4033942909 0 0) 0x7fedc8003b00 con 0x7fedd40b1f60 2024-03-29T11:22:02.071 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:02.091+0000 7fedd1ffb640 4 set_mon_vals no callback set 2024-03-29T11:22:02.071 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:02.092+0000 7fedbbfff640 1 -- 172.21.15.84:0/770339044 <== mon.2 v1:172.21.15.84:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2808139587 0 0) 0x7fedc8084be0 con 0x7fedd40b1f60 2024-03-29T11:22:02.071 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:02.092+0000 7fedbbfff640 10 client.4322.objecter ms_dispatch 0x7fedd4000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:22:02.071 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:02.092+0000 7fedbbfff640 3 client.4322.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:22:02.072 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:02.092+0000 7fedbbfff640 3 client.4322.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:22:02.072 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:02.092+0000 7fedd2ffd640 10 monclient: get_auth_request con 0x7fedc006ddc0 auth_method 0 2024-03-29T11:22:02.072 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:02.092+0000 7fedbbfff640 20 client.4322.objecter dump_active .. 0 homeless 2024-03-29T11:22:02.072 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:02.092+0000 7fedbbfff640 10 client.4322.objecter ms_handle_connect 0x7fedc006ddc0 2024-03-29T11:22:02.077 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:02.100+0000 7fedbbfff640 1 -- 172.21.15.84:0/770339044 <== mon.2 v1:172.21.15.84:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fedc8050540 con 0x7fedd40b1f60 2024-03-29T11:22:02.078 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:02.100+0000 7fedbbfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:22:02.078 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:02.100+0000 7fedbbfff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:22:02.405 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:02.428+0000 7fedd9e4a640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:22:02.405 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:02.428+0000 7fedd9e4a640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:22:02.405 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:02.428+0000 7fedd9e4a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-29T11:22:02.405 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:02.428+0000 7fedd9e4a640 1 -- 172.21.15.84:0/770339044 --> v1:172.21.15.84:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fedd4075f10 con 0x7fedd40b1f60 2024-03-29T11:22:02.406 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:02.430+0000 7fedbbfff640 1 -- 172.21.15.84:0/770339044 <== mon.2 v1:172.21.15.84:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4209477549) 0x7fedc8052470 con 0x7fedd40b1f60 2024-03-29T11:22:02.406 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:02.430+0000 7fedbbfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:22:02.406 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:02.430+0000 7fedbbfff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:22:02.407 INFO:teuthology.misc.health.smithi084.stdout: 2024-03-29T11:22:02.407 INFO:teuthology.misc.health.smithi084.stdout:{"epoch":10,"fsid":"2c3e002f-9eb0-4860-853b-4ef1ef562213","created":"2024-03-29T11:19:32.029169+0000","modified":"2024-03-29T11:19:34.200299+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:34.200299+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"5c39e276-2034-4716-bbc0-57a634eefa92","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"cd580e99-109f-4bf1-88c0-a5987bc8b9ed","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"11e60a5a-051d-453f-a190-66387cb35592","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"2c329fbe-bbc4-4030-bf87-d711cad04679","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f4213d4a-bd36-4433-91ae-d7e2b02481d3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"f0cf44ea-6055-48af-805b-aed858102ff9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"d895eaa0-cc0b-4ee4-9b5a-2dfcd89a60b7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"9bc5a7bc-8f52-4db9-96d6-de5e68d52eaa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:22:02.409 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:02.433+0000 7fedd9e4a640 10 librados: watch_flush enter 2024-03-29T11:22:02.411 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:02.434+0000 7fedd9e4a640 10 librados: watch_flush exit 2024-03-29T11:22:02.411 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:02.434+0000 7fedd9e4a640 20 client.4322.objecter shutdown clearing up homeless session... 2024-03-29T11:22:02.411 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:02.434+0000 7fedd9e4a640 10 client.4322.objecter successfully canceled tick 2024-03-29T11:22:02.412 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:02.435+0000 7fedd9e4a640 1 -- 172.21.15.84:0/770339044 >> v1:172.21.15.179:6816/2981134980 conn(0x7fedc006ddc0 legacy=0x7fedc0070280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:02.412 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:02.435+0000 7fedd9e4a640 10 monclient: shutdown 2024-03-29T11:22:02.412 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:02.435+0000 7fedd9e4a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:22:02.412 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:02.435+0000 7fedd9e4a640 1 -- 172.21.15.84:0/770339044 >> v1:172.21.15.84:6790/0 conn(0x7fedd40b1f60 legacy=0x7fedd408b5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:02.412 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:02.435+0000 7fedd9e4a640 1 -- 172.21.15.84:0/770339044 shutdown_connections 2024-03-29T11:22:02.412 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:02.435+0000 7fedd9e4a640 1 -- 172.21.15.84:0/770339044 >> 172.21.15.84:0/770339044 conn(0x7fedd4066a00 msgr2=0x7fedd4146a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:22:02.412 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:02.435+0000 7fedd9e4a640 1 -- 172.21.15.84:0/770339044 shutdown_connections 2024-03-29T11:22:02.413 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:02.435+0000 7fedd9e4a640 1 -- 172.21.15.84:0/770339044 wait complete. 2024-03-29T11:22:02.413 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:02.435+0000 7fedd9e4a640 1 librados: shutdown 2024-03-29T11:22:02.431 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:22:09.238 DEBUG:teuthology.orchestra.run.smithi084:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:22:09.385 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:09.408+0000 7f029a78c640 10 monclient: get_monmap_and_config 2024-03-29T11:22:09.385 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:09.408+0000 7f029a78c640 10 monclient: build_initial_monmap 2024-03-29T11:22:09.385 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:09.408+0000 7f029a78c640 10 monclient: monmap: 2024-03-29T11:22:09.385 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-29T11:22:09.385 INFO:teuthology.misc.health.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:22:09.385 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T11:22:09.409116+0000 2024-03-29T11:22:09.386 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T11:22:09.409116+0000 2024-03-29T11:22:09.386 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-29T11:22:09.386 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:22:09.386 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-29T11:22:09.386 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.noname-c 2024-03-29T11:22:09.386 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-b 2024-03-29T11:22:09.387 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:22:09.388 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:09.411+0000 7f029a78c640 1 Processor -- start 2024-03-29T11:22:09.388 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:09.411+0000 7f029a78c640 1 -- start start 2024-03-29T11:22:09.388 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:09.411+0000 7f029a78c640 10 monclient: init 2024-03-29T11:22:09.390 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:09.413+0000 7f029a78c640 10 monclient: _reopen_session rank -1 2024-03-29T11:22:09.390 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:09.413+0000 7f029a78c640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:22:09.390 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:09.413+0000 7f029a78c640 10 monclient(hunting): picked mon.noname-b con 0x7f02940b1f60 addr v1:172.21.15.84:6790/0 2024-03-29T11:22:09.390 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:09.413+0000 7f029a78c640 10 monclient(hunting): picked mon.noname-c con 0x7f0294071eb0 addr v1:172.21.15.179:6789/0 2024-03-29T11:22:09.390 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:09.413+0000 7f029a78c640 10 monclient(hunting): picked mon.noname-a con 0x7f0294143d10 addr v1:172.21.15.84:6789/0 2024-03-29T11:22:09.390 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:09.413+0000 7f029a78c640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f029406ef80 con 0x7f0294143d10 2024-03-29T11:22:09.390 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:09.413+0000 7f029a78c640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f029406f260 con 0x7f0294071eb0 2024-03-29T11:22:09.390 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:09.413+0000 7f029a78c640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f02940b40c0 con 0x7f02940b1f60 2024-03-29T11:22:09.390 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:09.413+0000 7f029a78c640 10 monclient(hunting): _renew_subs 2024-03-29T11:22:09.390 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:09.413+0000 7f029a78c640 10 monclient(hunting): authenticate will time out at 990.707489s 2024-03-29T11:22:09.391 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:09.414+0000 7f0293fff640 1 --1- >> v1:172.21.15.84:6790/0 conn(0x7f02940b1f60 0x7f0294143600 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.84:6790/0 says I am v1:172.21.15.84:60558/0 (socket says 172.21.15.84:60558) 2024-03-29T11:22:09.391 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:09.414+0000 7f0293fff640 1 -- 172.21.15.84:0/912991520 learned_addr learned my addr 172.21.15.84:0/912991520 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-29T11:22:09.392 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:09.416+0000 7f0292ffd640 1 -- 172.21.15.84:0/912991520 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1095097204 0 0) 0x7f02940b40c0 con 0x7f02940b1f60 2024-03-29T11:22:09.392 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:09.416+0000 7f0292ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:22:09.392 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:09.416+0000 7f0292ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:22:09.392 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:09.416+0000 7f0292ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:22:09.392 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:09.416+0000 7f0292ffd640 10 monclient(hunting): my global_id is 4325 2024-03-29T11:22:09.392 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:09.416+0000 7f0292ffd640 1 -- 172.21.15.84:0/912991520 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f027c003520 con 0x7f02940b1f60 2024-03-29T11:22:09.393 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:09.416+0000 7f0292ffd640 1 -- 172.21.15.84:0/912991520 <== mon.0 v1:172.21.15.84:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3247927050 0 0) 0x7f029406ef80 con 0x7f0294143d10 2024-03-29T11:22:09.393 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:09.416+0000 7f0292ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:22:09.393 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:09.416+0000 7f0292ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:22:09.393 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:09.416+0000 7f0292ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:22:09.393 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:09.416+0000 7f0292ffd640 10 monclient(hunting): my global_id is 4329 2024-03-29T11:22:09.393 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:09.416+0000 7f0292ffd640 1 -- 172.21.15.84:0/912991520 --> v1:172.21.15.84:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f02940b40c0 con 0x7f0294143d10 2024-03-29T11:22:09.393 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:09.416+0000 7f0292ffd640 1 -- 172.21.15.84:0/912991520 <== mon.0 v1:172.21.15.84:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4087618587 0 0) 0x7f02940b40c0 con 0x7f0294143d10 2024-03-29T11:22:09.394 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:09.416+0000 7f0292ffd640 1 -- 172.21.15.84:0/912991520 >> v1:172.21.15.84:6790/0 conn(0x7f02940b1f60 legacy=0x7f0294143600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:09.394 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:09.416+0000 7f0292ffd640 1 -- 172.21.15.84:0/912991520 >> v1:172.21.15.179:6789/0 conn(0x7f0294071eb0 legacy=0x7f0294146470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:22:09.394 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:09.416+0000 7f0292ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:22:09.394 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:09.416+0000 7f0292ffd640 1 monclient: found mon.noname-a 2024-03-29T11:22:09.394 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:09.416+0000 7f0292ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:22:09.394 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:09.416+0000 7f0292ffd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.84:6789/0 2024-03-29T11:22:09.394 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:09.416+0000 7f0292ffd640 1 -- 172.21.15.84:0/912991520 --> v1:172.21.15.84:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0294155d70 con 0x7f0294143d10 2024-03-29T11:22:09.394 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:09.416+0000 7f0292ffd640 10 monclient: _finish_auth 0 2024-03-29T11:22:09.394 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:09.416+0000 7f0292ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:22:09.394 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:09.416+0000 7f0292ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:22:09.394 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:09.416+0000 7f029a78c640 5 monclient: authenticate success, global_id 4329 2024-03-29T11:22:09.394 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:09.416+0000 7f029a78c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:22:09.394 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:09.416+0000 7f0292ffd640 1 -- 172.21.15.84:0/912991520 <== mon.0 v1:172.21.15.84:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4236868694 0 0) 0x7f0288001670 con 0x7f0294143d10 2024-03-29T11:22:09.395 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:09.417+0000 7f0292ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:09.395 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:09.417+0000 7f0292ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:22:09.395 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:09.417+0000 7f0292ffd640 10 monclient: dump: 2024-03-29T11:22:09.395 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T11:22:09.395 INFO:teuthology.misc.health.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:22:09.395 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T11:18:53.864025+0000 2024-03-29T11:22:09.395 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T11:18:53.864025+0000 2024-03-29T11:22:09.395 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:09.395 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:22:09.395 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T11:22:09.395 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T11:22:09.395 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T11:22:09.395 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:22:09.395 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:09.417+0000 7f029a78c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:22:09.396 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:09.417+0000 7f0292ffd640 1 -- 172.21.15.84:0/912991520 <== mon.0 v1:172.21.15.84:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0288001890 con 0x7f0294143d10 2024-03-29T11:22:09.396 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:09.417+0000 7f0292ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:22:09.396 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:09.417+0000 7f029a78c640 10 monclient: get_monmap_and_config success 2024-03-29T11:22:09.396 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:09.417+0000 7f029a78c640 4 set_mon_vals no callback set 2024-03-29T11:22:09.399 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:09.417+0000 7f0292ffd640 1 -- 172.21.15.84:0/912991520 <== mon.0 v1:172.21.15.84:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4236868694 0 0) 0x7f0288001c20 con 0x7f0294143d10 2024-03-29T11:22:09.399 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:09.417+0000 7f0292ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:09.399 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:09.417+0000 7f0292ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:22:09.399 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:09.417+0000 7f0292ffd640 10 monclient: dump: 2024-03-29T11:22:09.399 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T11:22:09.399 INFO:teuthology.misc.health.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:22:09.399 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T11:18:53.864025+0000 2024-03-29T11:22:09.399 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T11:18:53.864025+0000 2024-03-29T11:22:09.399 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:09.400 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:22:09.400 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T11:22:09.400 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T11:22:09.400 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T11:22:09.400 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:22:09.400 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:09.418+0000 7f029a78c640 10 monclient: shutdown 2024-03-29T11:22:09.400 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:09.418+0000 7f029a78c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:22:09.400 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:09.418+0000 7f029a78c640 1 -- 172.21.15.84:0/912991520 >> v1:172.21.15.84:6789/0 conn(0x7f0294143d10 legacy=0x7f0294146b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:09.400 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:09.418+0000 7f029a78c640 1 -- 172.21.15.84:0/912991520 shutdown_connections 2024-03-29T11:22:09.400 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:09.418+0000 7f029a78c640 1 -- 172.21.15.84:0/912991520 >> 172.21.15.84:0/912991520 conn(0x7f0294066a00 msgr2=0x7f0294066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:22:09.400 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:09.418+0000 7f029a78c640 1 -- 172.21.15.84:0/912991520 shutdown_connections 2024-03-29T11:22:09.400 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:09.418+0000 7f029a78c640 1 -- 172.21.15.84:0/912991520 wait complete. 2024-03-29T11:22:09.400 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:09.419+0000 7f029a78c640 10 monclient: build_initial_monmap 2024-03-29T11:22:09.401 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:09.419+0000 7f029a78c640 10 monclient: monmap: 2024-03-29T11:22:09.401 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-29T11:22:09.401 INFO:teuthology.misc.health.smithi084.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:22:09.401 INFO:teuthology.misc.health.smithi084.stderr:last_changed 0.000000 2024-03-29T11:22:09.401 INFO:teuthology.misc.health.smithi084.stderr:created 0.000000 2024-03-29T11:22:09.401 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-29T11:22:09.401 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:22:09.401 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-29T11:22:09.401 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.noname-b 2024-03-29T11:22:09.401 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-c 2024-03-29T11:22:09.401 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:22:09.401 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:09.420+0000 7f029a78c640 1 librados: starting msgr at 2024-03-29T11:22:09.401 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:09.420+0000 7f029a78c640 1 librados: starting objecter 2024-03-29T11:22:09.401 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:09.420+0000 7f029a78c640 1 Processor -- start 2024-03-29T11:22:09.401 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:09.420+0000 7f029a78c640 1 -- start start 2024-03-29T11:22:09.402 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:09.420+0000 7f029a78c640 1 librados: setting wanted keys 2024-03-29T11:22:09.402 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:09.421+0000 7f029a78c640 1 librados: calling monclient init 2024-03-29T11:22:09.402 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:09.421+0000 7f029a78c640 10 monclient: init 2024-03-29T11:22:09.402 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:09.422+0000 7f029a78c640 10 monclient: _reopen_session rank -1 2024-03-29T11:22:09.402 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:09.422+0000 7f029a78c640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:22:09.402 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:09.422+0000 7f029a78c640 10 monclient(hunting): picked mon.noname-a con 0x7f02940b1f60 addr v1:172.21.15.84:6789/0 2024-03-29T11:22:09.402 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:09.422+0000 7f029a78c640 10 monclient(hunting): picked mon.noname-b con 0x7f0294143d10 addr v1:172.21.15.179:6789/0 2024-03-29T11:22:09.402 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:09.422+0000 7f029a78c640 10 monclient(hunting): picked mon.noname-c con 0x7f0294136ca0 addr v1:172.21.15.84:6790/0 2024-03-29T11:22:09.402 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:09.422+0000 7f029a78c640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0294155580 con 0x7f02940b1f60 2024-03-29T11:22:09.402 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:09.422+0000 7f029a78c640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0294140b80 con 0x7f0294143d10 2024-03-29T11:22:09.402 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:09.422+0000 7f029a78c640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0294132740 con 0x7f0294136ca0 2024-03-29T11:22:09.402 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:09.422+0000 7f029a78c640 10 monclient(hunting): _renew_subs 2024-03-29T11:22:09.402 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:09.422+0000 7f029a78c640 10 monclient(hunting): authenticate will time out at 990.716825s 2024-03-29T11:22:09.404 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:09.424+0000 7f0298d29640 1 --1- >> v1:172.21.15.84:6790/0 conn(0x7f0294136ca0 0x7f0294132030 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.84:6790/0 says I am v1:172.21.15.84:33298/0 (socket says 172.21.15.84:33298) 2024-03-29T11:22:09.404 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:09.424+0000 7f0298d29640 1 -- 172.21.15.84:0/2308287021 learned_addr learned my addr 172.21.15.84:0/2308287021 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-29T11:22:09.404 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:09.424+0000 7f0290ff9640 10 client.?.objecter ms_handle_connect 0x7f0294136ca0 2024-03-29T11:22:09.404 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:09.424+0000 7f0290ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:22:09.404 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:09.424+0000 7f0290ff9640 1 -- 172.21.15.84:0/2308287021 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1149455097 0 0) 0x7f0294132740 con 0x7f0294136ca0 2024-03-29T11:22:09.404 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:09.424+0000 7f0290ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:22:09.405 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:09.424+0000 7f0290ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:22:09.405 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:09.424+0000 7f0290ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:22:09.405 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:09.424+0000 7f0290ff9640 10 monclient(hunting): my global_id is 4328 2024-03-29T11:22:09.405 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:09.424+0000 7f0290ff9640 1 -- 172.21.15.84:0/2308287021 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0280004eb0 con 0x7f0294136ca0 2024-03-29T11:22:09.405 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:09.425+0000 7f0290ff9640 1 -- 172.21.15.84:0/2308287021 <== mon.2 v1:172.21.15.84:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2261776385 0 0) 0x7f0280004eb0 con 0x7f0294136ca0 2024-03-29T11:22:09.405 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:09.425+0000 7f0290ff9640 1 -- 172.21.15.84:0/2308287021 --> v1:172.21.15.84:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0294132740 con 0x7f0294136ca0 2024-03-29T11:22:09.405 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:09.425+0000 7f0290ff9640 1 -- 172.21.15.84:0/2308287021 <== mon.2 v1:172.21.15.84:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4236868694 0 0) 0x7f0288001c60 con 0x7f0294136ca0 2024-03-29T11:22:09.405 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:09.425+0000 7f0290ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:09.405 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:09.425+0000 7f0290ff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:22:09.405 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:09.425+0000 7f0290ff9640 10 monclient(hunting): dump: 2024-03-29T11:22:09.405 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T11:22:09.405 INFO:teuthology.misc.health.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:22:09.405 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T11:18:53.864025+0000 2024-03-29T11:22:09.405 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T11:18:53.864025+0000 2024-03-29T11:22:09.406 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:09.406 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:22:09.406 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T11:22:09.406 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T11:22:09.406 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T11:22:09.406 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:22:09.406 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:09.425+0000 7f0290ff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:22:09.406 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:09.425+0000 7f0290ff9640 10 client.?.objecter ms_handle_connect 0x7f02940b1f60 2024-03-29T11:22:09.406 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:09.425+0000 7f0290ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:22:09.406 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:09.425+0000 7f0290ff9640 1 -- 172.21.15.84:0/2308287021 <== mon.2 v1:172.21.15.84:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 251350331 0 0) 0x7f0294132740 con 0x7f0294136ca0 2024-03-29T11:22:09.406 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:09.425+0000 7f0290ff9640 1 -- 172.21.15.84:0/2308287021 >> v1:172.21.15.179:6789/0 conn(0x7f0294143d10 legacy=0x7f0294131920 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:22:09.408 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:09.429+0000 7f0290ff9640 1 -- 172.21.15.84:0/2308287021 >> v1:172.21.15.84:6789/0 conn(0x7f02940b1f60 legacy=0x7f029406f620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:09.408 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:09.429+0000 7f0290ff9640 10 monclient: _finish_hunting 0 2024-03-29T11:22:09.408 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:09.429+0000 7f0290ff9640 1 monclient: found mon.c 2024-03-29T11:22:09.408 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:09.429+0000 7f0290ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:22:09.408 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:09.429+0000 7f0290ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-29T11:22:09.408 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:09.429+0000 7f0290ff9640 1 -- 172.21.15.84:0/2308287021 --> v1:172.21.15.84:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02941545b0 con 0x7f0294136ca0 2024-03-29T11:22:09.408 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:09.429+0000 7f0290ff9640 10 monclient: _finish_auth 0 2024-03-29T11:22:09.409 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:09.429+0000 7f0290ff9640 10 monclient: _check_auth_tickets 2024-03-29T11:22:09.409 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:09.429+0000 7f0290ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:22:09.409 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:09.429+0000 7f029a78c640 5 monclient: authenticate success, global_id 4328 2024-03-29T11:22:09.409 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:09.429+0000 7f029a78c640 10 monclient: _renew_subs 2024-03-29T11:22:09.409 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:09.429+0000 7f029a78c640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-29T11:22:09.409 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:09.429+0000 7f029a78c640 1 -- 172.21.15.84:0/2308287021 --> v1:172.21.15.84:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0294154890 con 0x7f0294136ca0 2024-03-29T11:22:09.409 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:09.429+0000 7f029a78c640 10 client.4328.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:22:09.409 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:09.429+0000 7f029a78c640 10 monclient: _renew_subs 2024-03-29T11:22:09.409 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:09.429+0000 7f029a78c640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-29T11:22:09.409 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:09.429+0000 7f029a78c640 1 -- 172.21.15.84:0/2308287021 --> v1:172.21.15.84:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f0294155270 con 0x7f0294136ca0 2024-03-29T11:22:09.409 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:09.429+0000 7f029a78c640 1 librados: init done 2024-03-29T11:22:09.410 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:09.432+0000 7f0290ff9640 1 -- 172.21.15.84:0/2308287021 <== mon.2 v1:172.21.15.84:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f02880038a0 con 0x7f0294136ca0 2024-03-29T11:22:09.410 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:09.432+0000 7f0290ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:22:09.410 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:09.433+0000 7f0290ff9640 1 -- 172.21.15.84:0/2308287021 <== mon.2 v1:172.21.15.84:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4236868694 0 0) 0x7f0288003cb0 con 0x7f0294136ca0 2024-03-29T11:22:09.411 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:09.433+0000 7f0290ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:09.411 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:09.433+0000 7f0290ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:22:09.411 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:09.433+0000 7f0290ff9640 10 monclient: dump: 2024-03-29T11:22:09.411 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T11:22:09.411 INFO:teuthology.misc.health.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:22:09.411 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T11:18:53.864025+0000 2024-03-29T11:22:09.411 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T11:18:53.864025+0000 2024-03-29T11:22:09.411 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:09.411 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:22:09.411 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T11:22:09.411 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T11:22:09.411 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T11:22:09.411 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:22:09.411 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:09.433+0000 7f0290ff9640 1 -- 172.21.15.84:0/2308287021 <== mon.2 v1:172.21.15.84:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4033942909 0 0) 0x7f0288003ed0 con 0x7f0294136ca0 2024-03-29T11:22:09.412 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:09.433+0000 7f02927fc640 4 set_mon_vals no callback set 2024-03-29T11:22:09.412 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:09.434+0000 7f0293fff640 10 monclient: get_auth_request con 0x7f028006dbd0 auth_method 0 2024-03-29T11:22:09.412 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:09.434+0000 7f0290ff9640 1 -- 172.21.15.84:0/2308287021 <== mon.2 v1:172.21.15.84:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2808139587 0 0) 0x7f0288084d00 con 0x7f0294136ca0 2024-03-29T11:22:09.412 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:09.434+0000 7f0290ff9640 10 client.4328.objecter ms_dispatch 0x7f0294000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:22:09.412 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:09.434+0000 7f0290ff9640 3 client.4328.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:22:09.412 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:09.434+0000 7f0290ff9640 3 client.4328.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:22:09.412 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:09.434+0000 7f0290ff9640 20 client.4328.objecter dump_active .. 0 homeless 2024-03-29T11:22:09.413 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:09.434+0000 7f0290ff9640 10 client.4328.objecter ms_handle_connect 0x7f028006dbd0 2024-03-29T11:22:09.413 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:09.435+0000 7f027a7fc640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:22:09.413 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:09.435+0000 7f027a7fc640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:22:09.413 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:09.435+0000 7f027a7fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-29T11:22:09.413 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:09.435+0000 7f027a7fc640 1 -- 172.21.15.84:0/2308287021 --> v1:172.21.15.84:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0274005080 con 0x7f0294136ca0 2024-03-29T11:22:09.424 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:09.446+0000 7f0290ff9640 1 -- 172.21.15.84:0/2308287021 <== mon.2 v1:172.21.15.84:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f0288050660 con 0x7f0294136ca0 2024-03-29T11:22:09.424 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:09.446+0000 7f0290ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:22:09.424 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:09.446+0000 7f0290ff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:22:09.744 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:09.768+0000 7f027a7fc640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:22:09.745 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:09.768+0000 7f027a7fc640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:22:09.745 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:09.768+0000 7f027a7fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-29T11:22:09.745 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:09.768+0000 7f027a7fc640 1 -- 172.21.15.84:0/2308287021 --> v1:172.21.15.84:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0274005330 con 0x7f0294136ca0 2024-03-29T11:22:09.746 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:09.770+0000 7f0290ff9640 1 -- 172.21.15.84:0/2308287021 <== mon.2 v1:172.21.15.84:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4209477549) 0x7f0288052590 con 0x7f0294136ca0 2024-03-29T11:22:09.746 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:09.770+0000 7f0290ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:22:09.747 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:09.770+0000 7f0290ff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:22:09.747 INFO:teuthology.misc.health.smithi084.stdout: 2024-03-29T11:22:09.747 INFO:teuthology.misc.health.smithi084.stdout:{"epoch":10,"fsid":"2c3e002f-9eb0-4860-853b-4ef1ef562213","created":"2024-03-29T11:19:32.029169+0000","modified":"2024-03-29T11:19:34.200299+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:34.200299+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"5c39e276-2034-4716-bbc0-57a634eefa92","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"cd580e99-109f-4bf1-88c0-a5987bc8b9ed","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"11e60a5a-051d-453f-a190-66387cb35592","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"2c329fbe-bbc4-4030-bf87-d711cad04679","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f4213d4a-bd36-4433-91ae-d7e2b02481d3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"f0cf44ea-6055-48af-805b-aed858102ff9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"d895eaa0-cc0b-4ee4-9b5a-2dfcd89a60b7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"9bc5a7bc-8f52-4db9-96d6-de5e68d52eaa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:22:09.751 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:09.775+0000 7f027a7fc640 10 librados: watch_flush enter 2024-03-29T11:22:09.751 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:09.775+0000 7f027a7fc640 10 librados: watch_flush exit 2024-03-29T11:22:09.751 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:09.775+0000 7f027a7fc640 20 client.4328.objecter shutdown clearing up homeless session... 2024-03-29T11:22:09.751 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:09.775+0000 7f027a7fc640 10 client.4328.objecter successfully canceled tick 2024-03-29T11:22:09.752 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:09.776+0000 7f027a7fc640 1 -- 172.21.15.84:0/2308287021 >> v1:172.21.15.179:6816/2981134980 conn(0x7f028006dbd0 legacy=0x7f0280070090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:09.753 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:09.776+0000 7f027a7fc640 10 monclient: shutdown 2024-03-29T11:22:09.753 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:09.776+0000 7f027a7fc640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:22:09.753 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:09.776+0000 7f027a7fc640 1 -- 172.21.15.84:0/2308287021 >> v1:172.21.15.84:6790/0 conn(0x7f0294136ca0 legacy=0x7f0294132030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:09.753 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:09.776+0000 7f027a7fc640 1 -- 172.21.15.84:0/2308287021 shutdown_connections 2024-03-29T11:22:09.753 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:09.776+0000 7f027a7fc640 1 -- 172.21.15.84:0/2308287021 >> 172.21.15.84:0/2308287021 conn(0x7f0294066a00 msgr2=0x7f02940b7bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:22:09.753 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:09.777+0000 7f027a7fc640 1 -- 172.21.15.84:0/2308287021 shutdown_connections 2024-03-29T11:22:09.753 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:09.777+0000 7f027a7fc640 1 -- 172.21.15.84:0/2308287021 wait complete. 2024-03-29T11:22:09.754 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:09.777+0000 7f027a7fc640 1 librados: shutdown 2024-03-29T11:22:09.773 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:22:16.579 DEBUG:teuthology.orchestra.run.smithi084:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:22:16.731 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:16.753+0000 7f797f8d9640 10 monclient: get_monmap_and_config 2024-03-29T11:22:16.731 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:16.753+0000 7f797f8d9640 10 monclient: build_initial_monmap 2024-03-29T11:22:16.731 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:16.753+0000 7f797f8d9640 10 monclient: monmap: 2024-03-29T11:22:16.731 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-29T11:22:16.731 INFO:teuthology.misc.health.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:22:16.731 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T11:22:16.754812+0000 2024-03-29T11:22:16.731 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T11:22:16.754812+0000 2024-03-29T11:22:16.731 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-29T11:22:16.731 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:22:16.731 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-29T11:22:16.731 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.noname-c 2024-03-29T11:22:16.731 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-b 2024-03-29T11:22:16.732 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:22:16.732 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:16.756+0000 7f797f8d9640 1 Processor -- start 2024-03-29T11:22:16.733 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:16.756+0000 7f797f8d9640 1 -- start start 2024-03-29T11:22:16.733 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:16.756+0000 7f797f8d9640 10 monclient: init 2024-03-29T11:22:16.733 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:16.757+0000 7f797f8d9640 10 monclient: _reopen_session rank -1 2024-03-29T11:22:16.733 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:16.757+0000 7f797f8d9640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:22:16.733 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:16.757+0000 7f797f8d9640 10 monclient(hunting): picked mon.noname-b con 0x7f79780a9ac0 addr v1:172.21.15.84:6790/0 2024-03-29T11:22:16.733 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:16.757+0000 7f797f8d9640 10 monclient(hunting): picked mon.noname-a con 0x7f797813ee90 addr v1:172.21.15.84:6789/0 2024-03-29T11:22:16.733 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:16.757+0000 7f797f8d9640 10 monclient(hunting): picked mon.noname-c con 0x7f79780af900 addr v1:172.21.15.179:6789/0 2024-03-29T11:22:16.734 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:16.757+0000 7f797f8d9640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f79780afd10 con 0x7f797813ee90 2024-03-29T11:22:16.734 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:16.757+0000 7f797f8d9640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f79780af6f0 con 0x7f79780af900 2024-03-29T11:22:16.734 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:16.757+0000 7f797f8d9640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f79780aac80 con 0x7f79780a9ac0 2024-03-29T11:22:16.734 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:16.757+0000 7f797f8d9640 10 monclient(hunting): _renew_subs 2024-03-29T11:22:16.734 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:16.757+0000 7f797f8d9640 10 monclient(hunting): authenticate will time out at 998.051467s 2024-03-29T11:22:16.739 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:16.763+0000 7f797de76640 1 --1- >> v1:172.21.15.179:6789/0 conn(0x7f79780af900 0x7f7978149b00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.179:6789/0 says I am v1:172.21.15.84:49859/0 (socket says 172.21.15.84:49859) 2024-03-29T11:22:16.740 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:16.763+0000 7f797ce74640 1 --1- >> v1:172.21.15.84:6789/0 conn(0x7f797813ee90 0x7f797813b5f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.84:6789/0 says I am v1:172.21.15.84:38493/0 (socket says 172.21.15.84:38493) 2024-03-29T11:22:16.740 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:16.763+0000 7f797ce74640 1 -- 172.21.15.84:0/858059657 learned_addr learned my addr 172.21.15.84:0/858059657 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-29T11:22:16.740 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:16.764+0000 7f796ffff640 1 -- 172.21.15.84:0/858059657 <== mon.0 v1:172.21.15.84:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4264346519 0 0) 0x7f79780afd10 con 0x7f797813ee90 2024-03-29T11:22:16.741 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:16.764+0000 7f796ffff640 10 monclient(hunting): _negotiate 2024-03-29T11:22:16.741 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:16.764+0000 7f796ffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:22:16.741 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:16.764+0000 7f796ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:22:16.741 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:16.764+0000 7f796ffff640 10 monclient(hunting): my global_id is 4335 2024-03-29T11:22:16.741 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:16.764+0000 7f796ffff640 1 -- 172.21.15.84:0/858059657 --> v1:172.21.15.84:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7964003520 con 0x7f797813ee90 2024-03-29T11:22:16.741 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:16.764+0000 7f796ffff640 1 -- 172.21.15.84:0/858059657 <== mon.0 v1:172.21.15.84:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 454011903 0 0) 0x7f7964003520 con 0x7f797813ee90 2024-03-29T11:22:16.741 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:16.764+0000 7f796ffff640 1 -- 172.21.15.84:0/858059657 >> v1:172.21.15.84:6790/0 conn(0x7f79780a9ac0 legacy=0x7f79780adfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:22:16.742 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:16.764+0000 7f796ffff640 1 -- 172.21.15.84:0/858059657 >> v1:172.21.15.179:6789/0 conn(0x7f79780af900 legacy=0x7f7978149b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:16.742 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:16.765+0000 7f796ffff640 10 monclient: _finish_hunting 0 2024-03-29T11:22:16.742 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:16.765+0000 7f796ffff640 1 monclient: found mon.noname-a 2024-03-29T11:22:16.742 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:16.765+0000 7f796ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:22:16.742 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:16.765+0000 7f796ffff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.84:6789/0 2024-03-29T11:22:16.742 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:16.765+0000 7f796ffff640 1 -- 172.21.15.84:0/858059657 --> v1:172.21.15.84:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79780aaf60 con 0x7f797813ee90 2024-03-29T11:22:16.742 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:16.765+0000 7f796ffff640 10 monclient: _finish_auth 0 2024-03-29T11:22:16.742 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:16.765+0000 7f796ffff640 10 monclient: _check_auth_tickets 2024-03-29T11:22:16.743 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:16.765+0000 7f796ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:22:16.743 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:16.765+0000 7f797f8d9640 5 monclient: authenticate success, global_id 4335 2024-03-29T11:22:16.743 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:16.765+0000 7f797f8d9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:22:16.743 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:16.765+0000 7f796ffff640 1 -- 172.21.15.84:0/858059657 <== mon.0 v1:172.21.15.84:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4236868694 0 0) 0x7f7974001690 con 0x7f797813ee90 2024-03-29T11:22:16.743 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:16.765+0000 7f796ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:16.743 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:16.765+0000 7f796ffff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:22:16.743 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:16.765+0000 7f796ffff640 10 monclient: dump: 2024-03-29T11:22:16.743 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T11:22:16.743 INFO:teuthology.misc.health.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:22:16.743 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T11:18:53.864025+0000 2024-03-29T11:22:16.743 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T11:18:53.864025+0000 2024-03-29T11:22:16.743 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:16.743 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:22:16.743 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T11:22:16.743 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T11:22:16.743 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T11:22:16.744 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:22:16.744 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:16.765+0000 7f797f8d9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:22:16.744 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:16.768+0000 7f796ffff640 1 -- 172.21.15.84:0/858059657 <== mon.0 v1:172.21.15.84:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7974001860 con 0x7f797813ee90 2024-03-29T11:22:16.744 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:16.768+0000 7f796ffff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:22:16.745 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:16.768+0000 7f797f8d9640 10 monclient: get_monmap_and_config success 2024-03-29T11:22:16.745 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:16.768+0000 7f797f8d9640 4 set_mon_vals no callback set 2024-03-29T11:22:16.745 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:16.769+0000 7f797f8d9640 10 monclient: shutdown 2024-03-29T11:22:16.746 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:16.769+0000 7f797f8d9640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:22:16.746 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:16.769+0000 7f797f8d9640 1 -- 172.21.15.84:0/858059657 >> v1:172.21.15.84:6789/0 conn(0x7f797813ee90 legacy=0x7f797813b5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:16.746 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:16.770+0000 7f797f8d9640 1 -- 172.21.15.84:0/858059657 shutdown_connections 2024-03-29T11:22:16.746 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:16.770+0000 7f797f8d9640 1 -- 172.21.15.84:0/858059657 >> 172.21.15.84:0/858059657 conn(0x7f79780b35e0 msgr2=0x7f79780b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:22:16.746 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:16.770+0000 7f797f8d9640 1 -- 172.21.15.84:0/858059657 shutdown_connections 2024-03-29T11:22:16.749 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:16.770+0000 7f797f8d9640 1 -- 172.21.15.84:0/858059657 wait complete. 2024-03-29T11:22:16.751 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:16.771+0000 7f797f8d9640 10 monclient: build_initial_monmap 2024-03-29T11:22:16.751 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:16.771+0000 7f797f8d9640 10 monclient: monmap: 2024-03-29T11:22:16.751 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-29T11:22:16.751 INFO:teuthology.misc.health.smithi084.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:22:16.751 INFO:teuthology.misc.health.smithi084.stderr:last_changed 0.000000 2024-03-29T11:22:16.752 INFO:teuthology.misc.health.smithi084.stderr:created 0.000000 2024-03-29T11:22:16.752 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-29T11:22:16.752 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:22:16.752 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-29T11:22:16.752 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.noname-b 2024-03-29T11:22:16.752 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-c 2024-03-29T11:22:16.752 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:22:16.752 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:16.772+0000 7f797f8d9640 1 librados: starting msgr at 2024-03-29T11:22:16.752 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:16.772+0000 7f797f8d9640 1 librados: starting objecter 2024-03-29T11:22:16.752 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:16.774+0000 7f797f8d9640 1 Processor -- start 2024-03-29T11:22:16.752 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:16.774+0000 7f797f8d9640 1 -- start start 2024-03-29T11:22:16.752 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:16.774+0000 7f797f8d9640 1 librados: setting wanted keys 2024-03-29T11:22:16.752 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:16.774+0000 7f797f8d9640 1 librados: calling monclient init 2024-03-29T11:22:16.752 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:16.774+0000 7f797f8d9640 10 monclient: init 2024-03-29T11:22:16.752 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:16.774+0000 7f797f8d9640 10 monclient: _reopen_session rank -1 2024-03-29T11:22:16.753 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:16.774+0000 7f797f8d9640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:22:16.753 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:16.774+0000 7f797f8d9640 10 monclient(hunting): picked mon.noname-a con 0x7f79780a9ac0 addr v1:172.21.15.84:6789/0 2024-03-29T11:22:16.753 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:16.774+0000 7f797f8d9640 10 monclient(hunting): picked mon.noname-b con 0x7f79780af900 addr v1:172.21.15.179:6789/0 2024-03-29T11:22:16.753 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:16.774+0000 7f797f8d9640 10 monclient(hunting): picked mon.noname-c con 0x7f797813ee90 addr v1:172.21.15.84:6790/0 2024-03-29T11:22:16.753 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:16.774+0000 7f797f8d9640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7978071840 con 0x7f79780a9ac0 2024-03-29T11:22:16.753 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:16.774+0000 7f797f8d9640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f797812b800 con 0x7f79780af900 2024-03-29T11:22:16.753 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:16.774+0000 7f797f8d9640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f797812bae0 con 0x7f797813ee90 2024-03-29T11:22:16.753 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:16.774+0000 7f797f8d9640 10 monclient(hunting): _renew_subs 2024-03-29T11:22:16.753 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:16.774+0000 7f797f8d9640 10 monclient(hunting): authenticate will time out at 998.069066s 2024-03-29T11:22:16.757 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:16.778+0000 7f797de76640 1 --1- >> v1:172.21.15.84:6790/0 conn(0x7f797813ee90 0x7f797812f5d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.84:6790/0 says I am v1:172.21.15.84:48161/0 (socket says 172.21.15.84:48161) 2024-03-29T11:22:16.757 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:16.778+0000 7f797de76640 1 -- 172.21.15.84:0/2172330446 learned_addr learned my addr 172.21.15.84:0/2172330446 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-29T11:22:16.757 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:16.779+0000 7f796dffb640 10 client.?.objecter ms_handle_connect 0x7f797813ee90 2024-03-29T11:22:16.757 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:16.779+0000 7f796dffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:22:16.757 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:16.779+0000 7f796dffb640 1 -- 172.21.15.84:0/2172330446 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3751165879 0 0) 0x7f797812bae0 con 0x7f797813ee90 2024-03-29T11:22:16.757 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:16.779+0000 7f796dffb640 10 monclient(hunting): _negotiate 2024-03-29T11:22:16.757 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:16.779+0000 7f796dffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:22:16.757 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:16.779+0000 7f796dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:22:16.757 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:16.779+0000 7f796dffb640 10 monclient(hunting): my global_id is 4331 2024-03-29T11:22:16.757 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:16.779+0000 7f796dffb640 1 -- 172.21.15.84:0/2172330446 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f795c0092c0 con 0x7f797813ee90 2024-03-29T11:22:16.757 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:16.779+0000 7f796dffb640 1 -- 172.21.15.84:0/2172330446 <== mon.2 v1:172.21.15.84:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2038257728 0 0) 0x7f795c0092c0 con 0x7f797813ee90 2024-03-29T11:22:16.757 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:16.780+0000 7f796dffb640 1 -- 172.21.15.84:0/2172330446 --> v1:172.21.15.84:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f797812bae0 con 0x7f797813ee90 2024-03-29T11:22:16.757 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:16.780+0000 7f796dffb640 1 -- 172.21.15.84:0/2172330446 <== mon.2 v1:172.21.15.84:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4236868694 0 0) 0x7f7970001cd0 con 0x7f797813ee90 2024-03-29T11:22:16.758 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:16.780+0000 7f796dffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:16.758 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:16.780+0000 7f796dffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:22:16.758 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:16.780+0000 7f796dffb640 10 monclient(hunting): dump: 2024-03-29T11:22:16.758 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T11:22:16.758 INFO:teuthology.misc.health.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:22:16.758 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T11:18:53.864025+0000 2024-03-29T11:22:16.758 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T11:18:53.864025+0000 2024-03-29T11:22:16.758 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:16.758 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:22:16.758 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T11:22:16.758 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T11:22:16.759 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T11:22:16.759 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:22:16.759 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:16.780+0000 7f796dffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:22:16.759 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:16.780+0000 7f796dffb640 10 client.?.objecter ms_handle_connect 0x7f79780a9ac0 2024-03-29T11:22:16.759 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:16.780+0000 7f796dffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:22:16.759 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:16.780+0000 7f796dffb640 1 -- 172.21.15.84:0/2172330446 <== mon.2 v1:172.21.15.84:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 109733199 0 0) 0x7f797812bae0 con 0x7f797813ee90 2024-03-29T11:22:16.759 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:16.780+0000 7f796dffb640 1 -- 172.21.15.84:0/2172330446 >> v1:172.21.15.179:6789/0 conn(0x7f79780af900 legacy=0x7f797812a730 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-29T11:22:16.759 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:16.780+0000 7f796dffb640 1 -- 172.21.15.84:0/2172330446 >> v1:172.21.15.84:6789/0 conn(0x7f79780a9ac0 legacy=0x7f79781252d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:16.759 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:16.780+0000 7f796dffb640 10 monclient: _finish_hunting 0 2024-03-29T11:22:16.759 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:16.780+0000 7f796dffb640 1 monclient: found mon.c 2024-03-29T11:22:16.759 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:16.780+0000 7f796dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:22:16.759 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:16.780+0000 7f796dffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-29T11:22:16.759 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:16.780+0000 7f796dffb640 1 -- 172.21.15.84:0/2172330446 --> v1:172.21.15.84:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f797812bdc0 con 0x7f797813ee90 2024-03-29T11:22:16.759 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:16.780+0000 7f796dffb640 10 monclient: _finish_auth 0 2024-03-29T11:22:16.759 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:16.780+0000 7f796dffb640 10 monclient: _check_auth_tickets 2024-03-29T11:22:16.760 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:16.780+0000 7f796dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:22:16.760 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:16.780+0000 7f796dffb640 1 -- 172.21.15.84:0/2172330446 <== mon.2 v1:172.21.15.84:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7970001e30 con 0x7f797813ee90 2024-03-29T11:22:16.760 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:16.780+0000 7f796dffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:22:16.760 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:16.780+0000 7f796dffb640 1 -- 172.21.15.84:0/2172330446 <== mon.2 v1:172.21.15.84:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4236868694 0 0) 0x7f7970004fb0 con 0x7f797813ee90 2024-03-29T11:22:16.760 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:16.780+0000 7f796dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:16.760 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:16.780+0000 7f796dffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:22:16.760 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:16.780+0000 7f796dffb640 10 monclient: dump: 2024-03-29T11:22:16.760 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T11:22:16.760 INFO:teuthology.misc.health.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:22:16.760 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T11:18:53.864025+0000 2024-03-29T11:22:16.760 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T11:18:53.864025+0000 2024-03-29T11:22:16.760 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:16.760 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:22:16.760 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T11:22:16.761 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T11:22:16.761 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T11:22:16.761 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:22:16.762 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:16.780+0000 7f797f8d9640 5 monclient: authenticate success, global_id 4331 2024-03-29T11:22:16.762 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:16.780+0000 7f797f8d9640 10 monclient: _renew_subs 2024-03-29T11:22:16.762 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:16.780+0000 7f797f8d9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-29T11:22:16.762 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:16.780+0000 7f797f8d9640 1 -- 172.21.15.84:0/2172330446 --> v1:172.21.15.84:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7978071320 con 0x7f797813ee90 2024-03-29T11:22:16.763 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:16.781+0000 7f797f8d9640 10 client.4331.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:22:16.763 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:16.781+0000 7f797f8d9640 10 monclient: _renew_subs 2024-03-29T11:22:16.763 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:16.781+0000 7f797f8d9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-29T11:22:16.763 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:16.781+0000 7f797f8d9640 1 -- 172.21.15.84:0/2172330446 --> v1:172.21.15.84:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f797812a240 con 0x7f797813ee90 2024-03-29T11:22:16.763 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:16.781+0000 7f797f8d9640 1 librados: init done 2024-03-29T11:22:16.763 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:16.782+0000 7f796f7fe640 4 set_mon_vals no callback set 2024-03-29T11:22:16.763 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:16.782+0000 7f797f8d9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:22:16.763 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:16.783+0000 7f797f8d9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:22:16.763 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:16.783+0000 7f797f8d9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-29T11:22:16.763 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:16.783+0000 7f797f8d9640 1 -- 172.21.15.84:0/2172330446 --> v1:172.21.15.84:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7958005080 con 0x7f797813ee90 2024-03-29T11:22:16.764 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:16.784+0000 7f796dffb640 1 -- 172.21.15.84:0/2172330446 <== mon.2 v1:172.21.15.84:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4033942909 0 0) 0x7f7970005110 con 0x7f797813ee90 2024-03-29T11:22:16.764 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:16.784+0000 7f796dffb640 1 -- 172.21.15.84:0/2172330446 <== mon.2 v1:172.21.15.84:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2808139587 0 0) 0x7f7970084f30 con 0x7f797813ee90 2024-03-29T11:22:16.764 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:16.784+0000 7f796dffb640 10 client.4331.objecter ms_dispatch 0x7f7978000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:22:16.764 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:16.784+0000 7f796dffb640 3 client.4331.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:22:16.764 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:16.784+0000 7f796dffb640 3 client.4331.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:22:16.764 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:16.784+0000 7f796dffb640 20 client.4331.objecter dump_active .. 0 homeless 2024-03-29T11:22:16.764 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:16.786+0000 7f797d675640 10 monclient: get_auth_request con 0x7f795c07ec90 auth_method 0 2024-03-29T11:22:16.764 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:16.788+0000 7f796dffb640 10 client.4331.objecter ms_handle_connect 0x7f795c07ec90 2024-03-29T11:22:16.768 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:16.792+0000 7f796dffb640 1 -- 172.21.15.84:0/2172330446 <== mon.2 v1:172.21.15.84:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f7970050640 con 0x7f797813ee90 2024-03-29T11:22:16.768 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:16.792+0000 7f796dffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:22:16.768 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:16.792+0000 7f796dffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:22:17.093 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:17.116+0000 7f797f8d9640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:22:17.093 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:17.116+0000 7f797f8d9640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:22:17.094 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:17.116+0000 7f797f8d9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-29T11:22:17.094 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:17.116+0000 7f797f8d9640 1 -- 172.21.15.84:0/2172330446 --> v1:172.21.15.84:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7958005330 con 0x7f797813ee90 2024-03-29T11:22:17.094 INFO:teuthology.misc.health.smithi084.stdout: 2024-03-29T11:22:17.094 INFO:teuthology.misc.health.smithi084.stdout:{"epoch":10,"fsid":"2c3e002f-9eb0-4860-853b-4ef1ef562213","created":"2024-03-29T11:19:32.029169+0000","modified":"2024-03-29T11:19:34.200299+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:34.200299+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"5c39e276-2034-4716-bbc0-57a634eefa92","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"cd580e99-109f-4bf1-88c0-a5987bc8b9ed","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"11e60a5a-051d-453f-a190-66387cb35592","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"2c329fbe-bbc4-4030-bf87-d711cad04679","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f4213d4a-bd36-4433-91ae-d7e2b02481d3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"f0cf44ea-6055-48af-805b-aed858102ff9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"d895eaa0-cc0b-4ee4-9b5a-2dfcd89a60b7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"9bc5a7bc-8f52-4db9-96d6-de5e68d52eaa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:22:17.095 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:17.118+0000 7f796dffb640 1 -- 172.21.15.84:0/2172330446 <== mon.2 v1:172.21.15.84:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4209477549) 0x7f7970052740 con 0x7f797813ee90 2024-03-29T11:22:17.095 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:17.118+0000 7f796dffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:22:17.095 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:17.118+0000 7f796dffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:22:17.098 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:17.122+0000 7f797f8d9640 10 librados: watch_flush enter 2024-03-29T11:22:17.098 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:17.122+0000 7f797f8d9640 10 librados: watch_flush exit 2024-03-29T11:22:17.098 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:17.122+0000 7f797f8d9640 20 client.4331.objecter shutdown clearing up homeless session... 2024-03-29T11:22:17.098 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:17.122+0000 7f797f8d9640 10 client.4331.objecter successfully canceled tick 2024-03-29T11:22:17.099 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:17.122+0000 7f797f8d9640 1 -- 172.21.15.84:0/2172330446 >> v1:172.21.15.179:6816/2981134980 conn(0x7f795c07ec90 legacy=0x7f795c081150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:17.099 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:17.122+0000 7f797f8d9640 10 monclient: shutdown 2024-03-29T11:22:17.099 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:17.122+0000 7f797f8d9640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:22:17.099 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:17.122+0000 7f797f8d9640 1 -- 172.21.15.84:0/2172330446 >> v1:172.21.15.84:6790/0 conn(0x7f797813ee90 legacy=0x7f797812f5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:17.099 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:17.122+0000 7f797f8d9640 1 -- 172.21.15.84:0/2172330446 shutdown_connections 2024-03-29T11:22:17.099 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:17.122+0000 7f797f8d9640 1 -- 172.21.15.84:0/2172330446 >> 172.21.15.84:0/2172330446 conn(0x7f79780b35e0 msgr2=0x7f7978132e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:22:17.099 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:17.122+0000 7f797f8d9640 1 -- 172.21.15.84:0/2172330446 shutdown_connections 2024-03-29T11:22:17.099 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:17.122+0000 7f797f8d9640 1 -- 172.21.15.84:0/2172330446 wait complete. 2024-03-29T11:22:17.100 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:17.124+0000 7f797f8d9640 1 librados: shutdown 2024-03-29T11:22:17.116 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:22:23.924 DEBUG:teuthology.orchestra.run.smithi084:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:22:24.072 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:24.093+0000 7f12a404e640 10 monclient: get_monmap_and_config 2024-03-29T11:22:24.072 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:24.093+0000 7f12a404e640 10 monclient: build_initial_monmap 2024-03-29T11:22:24.072 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:24.093+0000 7f12a404e640 10 monclient: monmap: 2024-03-29T11:22:24.072 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-29T11:22:24.072 INFO:teuthology.misc.health.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:22:24.072 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T11:22:24.094810+0000 2024-03-29T11:22:24.073 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T11:22:24.094810+0000 2024-03-29T11:22:24.073 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-29T11:22:24.073 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:22:24.073 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-29T11:22:24.073 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.noname-c 2024-03-29T11:22:24.073 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-b 2024-03-29T11:22:24.073 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:22:24.073 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:24.095+0000 7f12a404e640 1 Processor -- start 2024-03-29T11:22:24.080 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:24.096+0000 7f12a404e640 1 -- start start 2024-03-29T11:22:24.080 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:24.096+0000 7f12a404e640 10 monclient: init 2024-03-29T11:22:24.080 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:24.097+0000 7f12a404e640 10 monclient: _reopen_session rank -1 2024-03-29T11:22:24.081 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:24.097+0000 7f12a404e640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:22:24.081 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:24.097+0000 7f12a404e640 10 monclient(hunting): picked mon.noname-b con 0x7f129c0a9ac0 addr v1:172.21.15.84:6790/0 2024-03-29T11:22:24.081 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:24.097+0000 7f12a404e640 10 monclient(hunting): picked mon.noname-a con 0x7f129c13ee90 addr v1:172.21.15.84:6789/0 2024-03-29T11:22:24.081 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:24.097+0000 7f12a404e640 10 monclient(hunting): picked mon.noname-c con 0x7f129c0af900 addr v1:172.21.15.179:6789/0 2024-03-29T11:22:24.081 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:24.097+0000 7f12a404e640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f129c0afd10 con 0x7f129c13ee90 2024-03-29T11:22:24.081 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:24.097+0000 7f12a404e640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f129c0af6f0 con 0x7f129c0af900 2024-03-29T11:22:24.081 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:24.097+0000 7f12a404e640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f129c0aac80 con 0x7f129c0a9ac0 2024-03-29T11:22:24.081 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:24.097+0000 7f12a404e640 10 monclient(hunting): _renew_subs 2024-03-29T11:22:24.082 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:24.097+0000 7f12a404e640 10 monclient(hunting): authenticate will time out at 1005.391423s 2024-03-29T11:22:24.082 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:24.100+0000 7f12a15e9640 1 --1- >> v1:172.21.15.84:6789/0 conn(0x7f129c13ee90 0x7f129c13b5f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.84:6789/0 says I am v1:172.21.15.84:38279/0 (socket says 172.21.15.84:38279) 2024-03-29T11:22:24.082 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:24.100+0000 7f12a15e9640 1 -- 172.21.15.84:0/603746089 learned_addr learned my addr 172.21.15.84:0/603746089 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-29T11:22:24.082 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:24.100+0000 7f12a0de8640 1 -- 172.21.15.84:0/603746089 <== mon.0 v1:172.21.15.84:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 952287680 0 0) 0x7f129c0afd10 con 0x7f129c13ee90 2024-03-29T11:22:24.082 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:24.100+0000 7f12a0de8640 10 monclient(hunting): _negotiate 2024-03-29T11:22:24.082 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:24.100+0000 7f12a0de8640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:22:24.082 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:24.100+0000 7f12a0de8640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:22:24.082 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:24.100+0000 7f12a0de8640 10 monclient(hunting): my global_id is 4341 2024-03-29T11:22:24.083 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:24.100+0000 7f12a0de8640 1 -- 172.21.15.84:0/603746089 --> v1:172.21.15.84:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1288003520 con 0x7f129c13ee90 2024-03-29T11:22:24.083 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:24.101+0000 7f12a0de8640 1 -- 172.21.15.84:0/603746089 <== mon.0 v1:172.21.15.84:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1926293546 0 0) 0x7f1288003520 con 0x7f129c13ee90 2024-03-29T11:22:24.083 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:24.101+0000 7f12a0de8640 1 -- 172.21.15.84:0/603746089 >> v1:172.21.15.84:6790/0 conn(0x7f129c0a9ac0 legacy=0x7f129c0adfe0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:22:24.090 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:24.107+0000 7f12a0de8640 1 -- 172.21.15.84:0/603746089 >> v1:172.21.15.179:6789/0 conn(0x7f129c0af900 legacy=0x7f129c149b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:24.090 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:24.107+0000 7f12a0de8640 10 monclient: _finish_hunting 0 2024-03-29T11:22:24.090 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:24.107+0000 7f12a0de8640 1 monclient: found mon.noname-a 2024-03-29T11:22:24.091 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:24.109+0000 7f12a0de8640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:22:24.091 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:24.109+0000 7f12a0de8640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.84:6789/0 2024-03-29T11:22:24.091 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:24.109+0000 7f12a0de8640 1 -- 172.21.15.84:0/603746089 --> v1:172.21.15.84:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f129c0aaf60 con 0x7f129c13ee90 2024-03-29T11:22:24.091 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:24.109+0000 7f12a0de8640 10 monclient: _finish_auth 0 2024-03-29T11:22:24.091 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:24.109+0000 7f12a0de8640 10 monclient: _check_auth_tickets 2024-03-29T11:22:24.091 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:24.109+0000 7f12a0de8640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:22:24.091 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:24.109+0000 7f12a0de8640 1 -- 172.21.15.84:0/603746089 <== mon.0 v1:172.21.15.84:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4236868694 0 0) 0x7f1298001690 con 0x7f129c13ee90 2024-03-29T11:22:24.091 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:24.109+0000 7f12a0de8640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:24.091 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:24.109+0000 7f12a0de8640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:22:24.092 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:24.109+0000 7f12a0de8640 10 monclient: dump: 2024-03-29T11:22:24.092 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T11:22:24.092 INFO:teuthology.misc.health.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:22:24.092 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T11:18:53.864025+0000 2024-03-29T11:22:24.092 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T11:18:53.864025+0000 2024-03-29T11:22:24.092 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:24.092 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:22:24.092 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T11:22:24.093 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T11:22:24.093 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T11:22:24.093 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:22:24.093 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:24.109+0000 7f12a404e640 5 monclient: authenticate success, global_id 4341 2024-03-29T11:22:24.093 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:24.109+0000 7f12a404e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:22:24.093 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:24.110+0000 7f12a0de8640 1 -- 172.21.15.84:0/603746089 <== mon.0 v1:172.21.15.84:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1298001860 con 0x7f129c13ee90 2024-03-29T11:22:24.093 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:24.110+0000 7f12a0de8640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:22:24.093 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:24.110+0000 7f12a0de8640 1 -- 172.21.15.84:0/603746089 <== mon.0 v1:172.21.15.84:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4236868694 0 0) 0x7f1298001c70 con 0x7f129c13ee90 2024-03-29T11:22:24.094 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:24.110+0000 7f12a0de8640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:24.094 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:24.110+0000 7f12a0de8640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:22:24.094 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:24.110+0000 7f12a0de8640 10 monclient: dump: 2024-03-29T11:22:24.094 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T11:22:24.094 INFO:teuthology.misc.health.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:22:24.094 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T11:18:53.864025+0000 2024-03-29T11:22:24.094 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T11:18:53.864025+0000 2024-03-29T11:22:24.095 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:24.095 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:22:24.095 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T11:22:24.095 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T11:22:24.095 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T11:22:24.095 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:22:24.095 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:24.110+0000 7f12a404e640 10 monclient: get_monmap_and_config success 2024-03-29T11:22:24.095 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:24.110+0000 7f12a404e640 4 set_mon_vals no callback set 2024-03-29T11:22:24.096 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:24.111+0000 7f12a404e640 10 monclient: shutdown 2024-03-29T11:22:24.096 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:24.111+0000 7f12a404e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:22:24.096 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:24.111+0000 7f12a404e640 1 -- 172.21.15.84:0/603746089 >> v1:172.21.15.84:6789/0 conn(0x7f129c13ee90 legacy=0x7f129c13b5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:24.096 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:24.111+0000 7f12a404e640 1 -- 172.21.15.84:0/603746089 shutdown_connections 2024-03-29T11:22:24.096 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:24.111+0000 7f12a404e640 1 -- 172.21.15.84:0/603746089 >> 172.21.15.84:0/603746089 conn(0x7f129c0b35e0 msgr2=0x7f129c0b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:22:24.096 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:24.111+0000 7f12a404e640 1 -- 172.21.15.84:0/603746089 shutdown_connections 2024-03-29T11:22:24.096 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:24.111+0000 7f12a404e640 1 -- 172.21.15.84:0/603746089 wait complete. 2024-03-29T11:22:24.096 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:24.112+0000 7f12a404e640 10 monclient: build_initial_monmap 2024-03-29T11:22:24.097 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:24.112+0000 7f12a404e640 10 monclient: monmap: 2024-03-29T11:22:24.097 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-29T11:22:24.097 INFO:teuthology.misc.health.smithi084.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:22:24.097 INFO:teuthology.misc.health.smithi084.stderr:last_changed 0.000000 2024-03-29T11:22:24.097 INFO:teuthology.misc.health.smithi084.stderr:created 0.000000 2024-03-29T11:22:24.097 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-29T11:22:24.097 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:22:24.097 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-29T11:22:24.097 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.noname-b 2024-03-29T11:22:24.098 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-c 2024-03-29T11:22:24.098 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:22:24.099 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:24.113+0000 7f12a404e640 1 librados: starting msgr at 2024-03-29T11:22:24.099 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:24.113+0000 7f12a404e640 1 librados: starting objecter 2024-03-29T11:22:24.099 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:24.113+0000 7f12a404e640 1 Processor -- start 2024-03-29T11:22:24.099 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:24.116+0000 7f12a404e640 1 -- start start 2024-03-29T11:22:24.099 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:24.116+0000 7f12a404e640 1 librados: setting wanted keys 2024-03-29T11:22:24.100 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:24.116+0000 7f12a404e640 1 librados: calling monclient init 2024-03-29T11:22:24.100 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:24.116+0000 7f12a404e640 10 monclient: init 2024-03-29T11:22:24.100 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:24.117+0000 7f12a404e640 10 monclient: _reopen_session rank -1 2024-03-29T11:22:24.100 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:24.117+0000 7f12a404e640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:22:24.100 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:24.117+0000 7f12a404e640 10 monclient(hunting): picked mon.noname-c con 0x7f129c0a9ac0 addr v1:172.21.15.84:6790/0 2024-03-29T11:22:24.100 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:24.120+0000 7f12a404e640 10 monclient(hunting): picked mon.noname-b con 0x7f129c0af900 addr v1:172.21.15.179:6789/0 2024-03-29T11:22:24.102 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:24.121+0000 7f12a404e640 10 monclient(hunting): picked mon.noname-a con 0x7f129c13ee90 addr v1:172.21.15.84:6789/0 2024-03-29T11:22:24.103 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:24.121+0000 7f12a404e640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f129c12fa60 con 0x7f129c13ee90 2024-03-29T11:22:24.103 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:24.121+0000 7f12a404e640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f129c1400b0 con 0x7f129c0af900 2024-03-29T11:22:24.103 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:24.121+0000 7f12a404e640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f129c141310 con 0x7f129c0a9ac0 2024-03-29T11:22:24.103 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:24.121+0000 7f12a404e640 10 monclient(hunting): _renew_subs 2024-03-29T11:22:24.103 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:24.121+0000 7f12a404e640 10 monclient(hunting): authenticate will time out at 1005.415874s 2024-03-29T11:22:24.103 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:24.123+0000 7f12a15e9640 1 --1- >> v1:172.21.15.179:6789/0 conn(0x7f129c0af900 0x7f129c12c7d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.179:6789/0 says I am v1:172.21.15.84:44261/0 (socket says 172.21.15.84:44261) 2024-03-29T11:22:24.103 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:24.123+0000 7f12a15e9640 1 -- 172.21.15.84:0/3273541828 learned_addr learned my addr 172.21.15.84:0/3273541828 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-29T11:22:24.112 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:24.125+0000 7f12927fc640 10 client.?.objecter ms_handle_connect 0x7f129c0af900 2024-03-29T11:22:24.112 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:24.125+0000 7f12927fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:22:24.113 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:24.125+0000 7f12927fc640 10 client.?.objecter ms_handle_connect 0x7f129c13ee90 2024-03-29T11:22:24.113 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:24.125+0000 7f12927fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:22:24.113 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:24.125+0000 7f12927fc640 10 client.?.objecter ms_handle_connect 0x7f129c0a9ac0 2024-03-29T11:22:24.113 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:24.125+0000 7f12927fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:22:24.113 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:24.125+0000 7f12927fc640 1 -- 172.21.15.84:0/3273541828 <== mon.1 v1:172.21.15.179:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 804598636 0 0) 0x7f129c1400b0 con 0x7f129c0af900 2024-03-29T11:22:24.113 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:24.125+0000 7f12927fc640 10 monclient(hunting): _negotiate 2024-03-29T11:22:24.113 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:24.125+0000 7f12927fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:22:24.114 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:24.125+0000 7f12927fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:22:24.114 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:24.125+0000 7f12927fc640 10 monclient(hunting): my global_id is 4306 2024-03-29T11:22:24.114 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:24.125+0000 7f12927fc640 1 -- 172.21.15.84:0/3273541828 --> v1:172.21.15.179:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1280004eb0 con 0x7f129c0af900 2024-03-29T11:22:24.114 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:24.125+0000 7f12927fc640 1 -- 172.21.15.84:0/3273541828 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2999826725 0 0) 0x7f129c141310 con 0x7f129c0a9ac0 2024-03-29T11:22:24.114 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:24.125+0000 7f12927fc640 10 monclient(hunting): _negotiate 2024-03-29T11:22:24.114 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:24.125+0000 7f12927fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:22:24.114 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:24.125+0000 7f12927fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:22:24.114 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:24.125+0000 7f12927fc640 10 monclient(hunting): my global_id is 4337 2024-03-29T11:22:24.115 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:24.125+0000 7f12927fc640 1 -- 172.21.15.84:0/3273541828 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f129c1400b0 con 0x7f129c0a9ac0 2024-03-29T11:22:24.115 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:24.125+0000 7f12927fc640 1 -- 172.21.15.84:0/3273541828 <== mon.0 v1:172.21.15.84:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 254065947 0 0) 0x7f129c12fa60 con 0x7f129c13ee90 2024-03-29T11:22:24.115 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:24.126+0000 7f12927fc640 10 monclient(hunting): _negotiate 2024-03-29T11:22:24.115 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:24.126+0000 7f12927fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:22:24.115 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:24.126+0000 7f12927fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:22:24.115 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:24.126+0000 7f12927fc640 10 monclient(hunting): my global_id is 4344 2024-03-29T11:22:24.115 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:24.126+0000 7f12927fc640 1 -- 172.21.15.84:0/3273541828 --> v1:172.21.15.84:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f129c141310 con 0x7f129c13ee90 2024-03-29T11:22:24.115 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:24.129+0000 7f12927fc640 1 -- 172.21.15.84:0/3273541828 <== mon.2 v1:172.21.15.84:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2725838796 0 0) 0x7f129c1400b0 con 0x7f129c0a9ac0 2024-03-29T11:22:24.116 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:24.129+0000 7f12927fc640 1 -- 172.21.15.84:0/3273541828 --> v1:172.21.15.84:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f129c12fa60 con 0x7f129c0a9ac0 2024-03-29T11:22:24.116 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:24.129+0000 7f12927fc640 1 -- 172.21.15.84:0/3273541828 <== mon.1 v1:172.21.15.179:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1671399643 0 0) 0x7f1280004eb0 con 0x7f129c0af900 2024-03-29T11:22:24.116 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:24.129+0000 7f12927fc640 1 -- 172.21.15.84:0/3273541828 --> v1:172.21.15.179:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f129c1400b0 con 0x7f129c0af900 2024-03-29T11:22:24.116 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:24.129+0000 7f12927fc640 1 -- 172.21.15.84:0/3273541828 <== mon.0 v1:172.21.15.84:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2803538273 0 0) 0x7f129c141310 con 0x7f129c13ee90 2024-03-29T11:22:24.116 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:24.129+0000 7f12927fc640 1 -- 172.21.15.84:0/3273541828 --> v1:172.21.15.84:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1280004eb0 con 0x7f129c13ee90 2024-03-29T11:22:24.116 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:24.129+0000 7f12927fc640 1 -- 172.21.15.84:0/3273541828 <== mon.2 v1:172.21.15.84:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4236868694 0 0) 0x7f128c001830 con 0x7f129c0a9ac0 2024-03-29T11:22:24.116 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:24.129+0000 7f12927fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:24.116 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:24.129+0000 7f12927fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:22:24.117 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:24.129+0000 7f12927fc640 10 monclient(hunting): dump: 2024-03-29T11:22:24.117 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T11:22:24.117 INFO:teuthology.misc.health.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:22:24.117 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T11:18:53.864025+0000 2024-03-29T11:22:24.117 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T11:18:53.864025+0000 2024-03-29T11:22:24.117 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:24.117 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:22:24.117 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T11:22:24.118 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T11:22:24.118 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T11:22:24.118 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:22:24.118 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:24.129+0000 7f12927fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:22:24.118 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:24.129+0000 7f12927fc640 1 -- 172.21.15.84:0/3273541828 <== mon.1 v1:172.21.15.179:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4236868694 0 0) 0x7f1298001bb0 con 0x7f129c0af900 2024-03-29T11:22:24.118 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:24.129+0000 7f12927fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:24.118 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:24.129+0000 7f12927fc640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:22:24.118 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:24.129+0000 7f12927fc640 10 monclient(hunting): dump: 2024-03-29T11:22:24.119 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T11:22:24.119 INFO:teuthology.misc.health.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:22:24.119 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T11:18:53.864025+0000 2024-03-29T11:22:24.119 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T11:18:53.864025+0000 2024-03-29T11:22:24.119 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:24.119 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:22:24.119 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T11:22:24.119 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T11:22:24.120 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T11:22:24.120 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:22:24.120 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:24.129+0000 7f12927fc640 1 -- 172.21.15.84:0/3273541828 <== mon.0 v1:172.21.15.84:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4236868694 0 0) 0x7f1294001aa0 con 0x7f129c13ee90 2024-03-29T11:22:24.120 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:24.129+0000 7f12927fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:24.120 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:24.129+0000 7f12927fc640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:22:24.120 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:24.129+0000 7f12927fc640 10 monclient(hunting): dump: 2024-03-29T11:22:24.120 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T11:22:24.120 INFO:teuthology.misc.health.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:22:24.120 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T11:18:53.864025+0000 2024-03-29T11:22:24.121 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T11:18:53.864025+0000 2024-03-29T11:22:24.121 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:24.121 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:22:24.121 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T11:22:24.121 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T11:22:24.121 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T11:22:24.121 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:22:24.121 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:24.130+0000 7f12927fc640 1 -- 172.21.15.84:0/3273541828 <== mon.0 v1:172.21.15.84:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1820261918 0 0) 0x7f1280004eb0 con 0x7f129c13ee90 2024-03-29T11:22:24.122 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:24.130+0000 7f12927fc640 1 -- 172.21.15.84:0/3273541828 >> v1:172.21.15.84:6790/0 conn(0x7f129c0a9ac0 legacy=0x7f129c13d710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:24.122 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:24.130+0000 7f12927fc640 1 -- 172.21.15.84:0/3273541828 >> v1:172.21.15.179:6789/0 conn(0x7f129c0af900 legacy=0x7f129c12c7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:24.122 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:24.130+0000 7f12927fc640 10 monclient: _finish_hunting 0 2024-03-29T11:22:24.122 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:24.130+0000 7f12927fc640 1 monclient: found mon.a 2024-03-29T11:22:24.122 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:24.130+0000 7f12927fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:22:24.122 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:24.130+0000 7f12927fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.84:6789/0 2024-03-29T11:22:24.122 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:24.130+0000 7f12927fc640 1 -- 172.21.15.84:0/3273541828 --> v1:172.21.15.84:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f129c141f20 con 0x7f129c13ee90 2024-03-29T11:22:24.123 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:24.130+0000 7f12927fc640 10 monclient: _finish_auth 0 2024-03-29T11:22:24.123 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:24.130+0000 7f12927fc640 10 monclient: _check_auth_tickets 2024-03-29T11:22:24.123 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:24.130+0000 7f12927fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:22:24.123 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:24.130+0000 7f12a404e640 5 monclient: authenticate success, global_id 4344 2024-03-29T11:22:24.123 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:24.130+0000 7f12a404e640 10 monclient: _renew_subs 2024-03-29T11:22:24.123 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:24.130+0000 7f12a404e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.84:6789/0 2024-03-29T11:22:24.123 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:24.130+0000 7f12a404e640 1 -- 172.21.15.84:0/3273541828 --> v1:172.21.15.84:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f129c142200 con 0x7f129c13ee90 2024-03-29T11:22:24.123 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:24.130+0000 7f12a404e640 10 client.4344.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:22:24.124 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:24.130+0000 7f12a404e640 10 monclient: _renew_subs 2024-03-29T11:22:24.124 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:24.130+0000 7f12a404e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.84:6789/0 2024-03-29T11:22:24.124 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:24.130+0000 7f12a404e640 1 -- 172.21.15.84:0/3273541828 --> v1:172.21.15.84:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f129c11b1c0 con 0x7f129c13ee90 2024-03-29T11:22:24.124 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:24.130+0000 7f12a404e640 1 librados: init done 2024-03-29T11:22:24.124 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:24.131+0000 7f12927fc640 1 -- 172.21.15.84:0/3273541828 <== mon.0 v1:172.21.15.84:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1294001fb0 con 0x7f129c13ee90 2024-03-29T11:22:24.124 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:24.131+0000 7f12927fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:22:24.124 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:24.131+0000 7f12927fc640 1 -- 172.21.15.84:0/3273541828 <== mon.0 v1:172.21.15.84:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4236868694 0 0) 0x7f1294003b10 con 0x7f129c13ee90 2024-03-29T11:22:24.124 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:24.131+0000 7f12927fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:24.125 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:24.131+0000 7f12927fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:22:24.125 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:24.131+0000 7f12927fc640 10 monclient: dump: 2024-03-29T11:22:24.125 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T11:22:24.125 INFO:teuthology.misc.health.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:22:24.125 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T11:18:53.864025+0000 2024-03-29T11:22:24.125 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T11:18:53.864025+0000 2024-03-29T11:22:24.125 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:24.125 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:22:24.126 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T11:22:24.126 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T11:22:24.126 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T11:22:24.126 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:22:24.126 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:24.132+0000 7f1293fff640 4 set_mon_vals no callback set 2024-03-29T11:22:24.128 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:24.136+0000 7f1287fff640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:22:24.128 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:24.136+0000 7f1287fff640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:22:24.128 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:24.136+0000 7f1287fff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.84:6789/0 2024-03-29T11:22:24.128 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:24.136+0000 7f1287fff640 1 -- 172.21.15.84:0/3273541828 --> v1:172.21.15.84:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f127c005080 con 0x7f129c13ee90 2024-03-29T11:22:24.129 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:24.137+0000 7f12927fc640 1 -- 172.21.15.84:0/3273541828 <== mon.0 v1:172.21.15.84:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4033942909 0 0) 0x7f1294019ae0 con 0x7f129c13ee90 2024-03-29T11:22:24.129 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:24.138+0000 7f12927fc640 1 -- 172.21.15.84:0/3273541828 <== mon.0 v1:172.21.15.84:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2808139587 0 0) 0x7f12940838b0 con 0x7f129c13ee90 2024-03-29T11:22:24.129 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:24.138+0000 7f12927fc640 10 client.4344.objecter ms_dispatch 0x7f129c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:22:24.129 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:24.138+0000 7f12927fc640 3 client.4344.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:22:24.129 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:24.138+0000 7f12927fc640 3 client.4344.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:22:24.129 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:24.138+0000 7f12927fc640 20 client.4344.objecter dump_active .. 0 homeless 2024-03-29T11:22:24.129 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:24.140+0000 7f12a1dea640 10 monclient: get_auth_request con 0x7f128006e340 auth_method 0 2024-03-29T11:22:24.129 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:24.146+0000 7f12927fc640 10 client.4344.objecter ms_handle_connect 0x7f128006e340 2024-03-29T11:22:24.129 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:24.148+0000 7f12927fc640 1 -- 172.21.15.84:0/3273541828 <== mon.0 v1:172.21.15.84:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f129404f190 con 0x7f129c13ee90 2024-03-29T11:22:24.130 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:24.148+0000 7f12927fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:22:24.130 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:24.148+0000 7f12927fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:22:24.447 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:24.471+0000 7f1287fff640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:22:24.447 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:24.471+0000 7f1287fff640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:22:24.447 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:24.471+0000 7f1287fff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.84:6789/0 2024-03-29T11:22:24.447 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:24.471+0000 7f1287fff640 1 -- 172.21.15.84:0/3273541828 --> v1:172.21.15.84:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f127c005330 con 0x7f129c13ee90 2024-03-29T11:22:24.448 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:24.472+0000 7f12927fc640 1 -- 172.21.15.84:0/3273541828 <== mon.0 v1:172.21.15.84:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4209477549) 0x7f12940510c0 con 0x7f129c13ee90 2024-03-29T11:22:24.448 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:24.472+0000 7f12927fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:22:24.448 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:24.472+0000 7f12927fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:22:24.449 INFO:teuthology.misc.health.smithi084.stdout: 2024-03-29T11:22:24.449 INFO:teuthology.misc.health.smithi084.stdout:{"epoch":10,"fsid":"2c3e002f-9eb0-4860-853b-4ef1ef562213","created":"2024-03-29T11:19:32.029169+0000","modified":"2024-03-29T11:19:34.200299+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:34.200299+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"5c39e276-2034-4716-bbc0-57a634eefa92","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"cd580e99-109f-4bf1-88c0-a5987bc8b9ed","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"11e60a5a-051d-453f-a190-66387cb35592","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"2c329fbe-bbc4-4030-bf87-d711cad04679","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f4213d4a-bd36-4433-91ae-d7e2b02481d3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"f0cf44ea-6055-48af-805b-aed858102ff9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"d895eaa0-cc0b-4ee4-9b5a-2dfcd89a60b7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"9bc5a7bc-8f52-4db9-96d6-de5e68d52eaa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:22:24.452 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:24.476+0000 7f1287fff640 10 librados: watch_flush enter 2024-03-29T11:22:24.452 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:24.476+0000 7f1287fff640 10 librados: watch_flush exit 2024-03-29T11:22:24.452 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:24.476+0000 7f1287fff640 20 client.4344.objecter shutdown clearing up homeless session... 2024-03-29T11:22:24.452 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:24.476+0000 7f1287fff640 10 client.4344.objecter successfully canceled tick 2024-03-29T11:22:24.452 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:24.476+0000 7f1287fff640 1 -- 172.21.15.84:0/3273541828 >> v1:172.21.15.179:6816/2981134980 conn(0x7f128006e340 legacy=0x7f1280070800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:24.452 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:24.476+0000 7f1287fff640 10 monclient: shutdown 2024-03-29T11:22:24.452 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:24.476+0000 7f1287fff640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:22:24.452 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:24.476+0000 7f1287fff640 1 -- 172.21.15.84:0/3273541828 >> v1:172.21.15.84:6789/0 conn(0x7f129c13ee90 legacy=0x7f129c140c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:24.453 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:24.477+0000 7f1287fff640 1 -- 172.21.15.84:0/3273541828 shutdown_connections 2024-03-29T11:22:24.453 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:24.477+0000 7f1287fff640 1 -- 172.21.15.84:0/3273541828 >> 172.21.15.84:0/3273541828 conn(0x7f129c0b35e0 msgr2=0x7f129c0acef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:22:24.453 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:24.477+0000 7f1287fff640 1 -- 172.21.15.84:0/3273541828 shutdown_connections 2024-03-29T11:22:24.453 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:24.477+0000 7f1287fff640 1 -- 172.21.15.84:0/3273541828 wait complete. 2024-03-29T11:22:24.453 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:24.477+0000 7f1287fff640 1 librados: shutdown 2024-03-29T11:22:24.472 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:22:31.278 DEBUG:teuthology.orchestra.run.smithi084:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:22:31.426 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:31.450+0000 7f666764e640 10 monclient: get_monmap_and_config 2024-03-29T11:22:31.427 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:31.450+0000 7f666764e640 10 monclient: build_initial_monmap 2024-03-29T11:22:31.427 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:31.450+0000 7f666764e640 10 monclient: monmap: 2024-03-29T11:22:31.427 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-29T11:22:31.427 INFO:teuthology.misc.health.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:22:31.427 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T11:22:31.451362+0000 2024-03-29T11:22:31.428 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T11:22:31.451362+0000 2024-03-29T11:22:31.428 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-29T11:22:31.428 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:22:31.428 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-29T11:22:31.428 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.noname-c 2024-03-29T11:22:31.428 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-b 2024-03-29T11:22:31.429 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:22:31.429 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:31.451+0000 7f666764e640 1 Processor -- start 2024-03-29T11:22:31.429 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:31.451+0000 7f666764e640 1 -- start start 2024-03-29T11:22:31.429 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:31.451+0000 7f666764e640 10 monclient: init 2024-03-29T11:22:31.429 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:31.452+0000 7f666764e640 10 monclient: _reopen_session rank -1 2024-03-29T11:22:31.429 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:31.452+0000 7f666764e640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:22:31.429 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:31.452+0000 7f666764e640 10 monclient(hunting): picked mon.noname-c con 0x7f66600b4790 addr v1:172.21.15.179:6789/0 2024-03-29T11:22:31.430 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:31.452+0000 7f666764e640 10 monclient(hunting): picked mon.noname-a con 0x7f66600aea90 addr v1:172.21.15.84:6789/0 2024-03-29T11:22:31.430 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:31.452+0000 7f666764e640 10 monclient(hunting): picked mon.noname-b con 0x7f66600b39f0 addr v1:172.21.15.84:6790/0 2024-03-29T11:22:31.430 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:31.452+0000 7f666764e640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f66600b0c60 con 0x7f66600aea90 2024-03-29T11:22:31.430 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:31.452+0000 7f666764e640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f66600af020 con 0x7f66600b4790 2024-03-29T11:22:31.430 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:31.452+0000 7f666764e640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f666013fbd0 con 0x7f66600b39f0 2024-03-29T11:22:31.430 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:31.452+0000 7f666764e640 10 monclient(hunting): _renew_subs 2024-03-29T11:22:31.430 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:31.452+0000 7f666764e640 10 monclient(hunting): authenticate will time out at 1012.746717s 2024-03-29T11:22:31.432 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:31.456+0000 7f6665beb640 1 --1- >> v1:172.21.15.84:6790/0 conn(0x7f66600b39f0 0x7f666013f4c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.84:6790/0 says I am v1:172.21.15.84:33411/0 (socket says 172.21.15.84:33411) 2024-03-29T11:22:31.432 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:31.456+0000 7f6665beb640 1 -- 172.21.15.84:0/2799183543 learned_addr learned my addr 172.21.15.84:0/2799183543 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-29T11:22:31.436 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:31.458+0000 7f6657fff640 1 -- 172.21.15.84:0/2799183543 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2874189012 0 0) 0x7f666013fbd0 con 0x7f66600b39f0 2024-03-29T11:22:31.436 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:31.458+0000 7f6657fff640 10 monclient(hunting): _negotiate 2024-03-29T11:22:31.436 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:31.458+0000 7f6657fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:22:31.436 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:31.458+0000 7f6657fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:22:31.436 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:31.458+0000 7f6657fff640 10 monclient(hunting): my global_id is 4340 2024-03-29T11:22:31.436 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:31.459+0000 7f6657fff640 1 -- 172.21.15.84:0/2799183543 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f664c003520 con 0x7f66600b39f0 2024-03-29T11:22:31.436 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:31.459+0000 7f6657fff640 1 -- 172.21.15.84:0/2799183543 <== mon.1 v1:172.21.15.179:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1274019388 0 0) 0x7f66600af020 con 0x7f66600b4790 2024-03-29T11:22:31.436 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:31.459+0000 7f6657fff640 10 monclient(hunting): _negotiate 2024-03-29T11:22:31.436 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:31.459+0000 7f6657fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:22:31.436 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:31.459+0000 7f6657fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:22:31.436 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:31.459+0000 7f6657fff640 10 monclient(hunting): my global_id is 4309 2024-03-29T11:22:31.437 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:31.459+0000 7f6657fff640 1 -- 172.21.15.84:0/2799183543 --> v1:172.21.15.179:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f666013fbd0 con 0x7f66600b4790 2024-03-29T11:22:31.437 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:31.459+0000 7f6657fff640 1 -- 172.21.15.84:0/2799183543 <== mon.1 v1:172.21.15.179:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3367841423 0 0) 0x7f666013fbd0 con 0x7f66600b4790 2024-03-29T11:22:31.437 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:31.459+0000 7f6657fff640 1 -- 172.21.15.84:0/2799183543 >> v1:172.21.15.84:6790/0 conn(0x7f66600b39f0 legacy=0x7f666013f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:31.437 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:31.459+0000 7f6657fff640 1 -- 172.21.15.84:0/2799183543 >> v1:172.21.15.84:6789/0 conn(0x7f66600aea90 legacy=0x7f66600af540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:31.437 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:31.461+0000 7f6657fff640 10 monclient: _finish_hunting 0 2024-03-29T11:22:31.437 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:31.461+0000 7f6657fff640 1 monclient: found mon.noname-c 2024-03-29T11:22:31.437 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:31.461+0000 7f6657fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:22:31.437 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:31.461+0000 7f6657fff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.179:6789/0 2024-03-29T11:22:31.437 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:31.461+0000 7f6657fff640 1 -- 172.21.15.84:0/2799183543 --> v1:172.21.15.179:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f666013fd60 con 0x7f66600b4790 2024-03-29T11:22:31.438 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:31.461+0000 7f6657fff640 10 monclient: _finish_auth 0 2024-03-29T11:22:31.438 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:31.461+0000 7f6657fff640 10 monclient: _check_auth_tickets 2024-03-29T11:22:31.438 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:31.461+0000 7f6657fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:22:31.438 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:31.461+0000 7f6657fff640 1 -- 172.21.15.84:0/2799183543 <== mon.1 v1:172.21.15.179:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4236868694 0 0) 0x7f66580018a0 con 0x7f66600b4790 2024-03-29T11:22:31.438 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:31.461+0000 7f6657fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:31.438 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:31.461+0000 7f6657fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:22:31.438 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:31.461+0000 7f6657fff640 10 monclient: dump: 2024-03-29T11:22:31.438 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T11:22:31.438 INFO:teuthology.misc.health.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:22:31.438 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T11:18:53.864025+0000 2024-03-29T11:22:31.438 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T11:18:53.864025+0000 2024-03-29T11:22:31.438 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:31.438 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:22:31.438 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T11:22:31.439 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T11:22:31.439 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T11:22:31.439 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:22:31.439 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:31.461+0000 7f666764e640 5 monclient: authenticate success, global_id 4309 2024-03-29T11:22:31.439 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:31.461+0000 7f666764e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:22:31.440 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:31.463+0000 7f6657fff640 1 -- 172.21.15.84:0/2799183543 <== mon.1 v1:172.21.15.179:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6658001a80 con 0x7f66600b4790 2024-03-29T11:22:31.440 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:31.463+0000 7f6657fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:22:31.440 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:31.463+0000 7f6657fff640 1 -- 172.21.15.84:0/2799183543 <== mon.1 v1:172.21.15.179:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4236868694 0 0) 0x7f6658001ed0 con 0x7f66600b4790 2024-03-29T11:22:31.440 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:31.463+0000 7f6657fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:31.440 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:31.463+0000 7f6657fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:22:31.440 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:31.463+0000 7f6657fff640 10 monclient: dump: 2024-03-29T11:22:31.440 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T11:22:31.440 INFO:teuthology.misc.health.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:22:31.440 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T11:18:53.864025+0000 2024-03-29T11:22:31.440 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T11:18:53.864025+0000 2024-03-29T11:22:31.440 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:31.440 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:22:31.440 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T11:22:31.440 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T11:22:31.440 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T11:22:31.440 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:22:31.441 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:31.463+0000 7f666764e640 10 monclient: get_monmap_and_config success 2024-03-29T11:22:31.441 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:31.463+0000 7f666764e640 4 set_mon_vals no callback set 2024-03-29T11:22:31.441 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:31.464+0000 7f666764e640 10 monclient: shutdown 2024-03-29T11:22:31.441 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:31.464+0000 7f666764e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:22:31.441 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:31.464+0000 7f666764e640 1 -- 172.21.15.84:0/2799183543 >> v1:172.21.15.179:6789/0 conn(0x7f66600b4790 legacy=0x7f66600b1080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:31.441 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:31.464+0000 7f666764e640 1 -- 172.21.15.84:0/2799183543 shutdown_connections 2024-03-29T11:22:31.441 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:31.464+0000 7f666764e640 1 -- 172.21.15.84:0/2799183543 >> 172.21.15.84:0/2799183543 conn(0x7f6660066a00 msgr2=0x7f6660066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:22:31.441 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:31.464+0000 7f666764e640 1 -- 172.21.15.84:0/2799183543 shutdown_connections 2024-03-29T11:22:31.441 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:31.464+0000 7f666764e640 1 -- 172.21.15.84:0/2799183543 wait complete. 2024-03-29T11:22:31.442 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:31.465+0000 7f666764e640 10 monclient: build_initial_monmap 2024-03-29T11:22:31.442 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:31.465+0000 7f666764e640 10 monclient: monmap: 2024-03-29T11:22:31.442 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-29T11:22:31.443 INFO:teuthology.misc.health.smithi084.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:22:31.443 INFO:teuthology.misc.health.smithi084.stderr:last_changed 0.000000 2024-03-29T11:22:31.443 INFO:teuthology.misc.health.smithi084.stderr:created 0.000000 2024-03-29T11:22:31.443 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-29T11:22:31.443 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:22:31.443 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-29T11:22:31.444 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.noname-b 2024-03-29T11:22:31.444 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-c 2024-03-29T11:22:31.444 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:22:31.444 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:31.465+0000 7f666764e640 1 librados: starting msgr at 2024-03-29T11:22:31.444 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:31.465+0000 7f666764e640 1 librados: starting objecter 2024-03-29T11:22:31.444 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:31.466+0000 7f666764e640 1 Processor -- start 2024-03-29T11:22:31.445 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:31.466+0000 7f666764e640 1 -- start start 2024-03-29T11:22:31.445 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:31.466+0000 7f666764e640 1 librados: setting wanted keys 2024-03-29T11:22:31.445 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:31.466+0000 7f666764e640 1 librados: calling monclient init 2024-03-29T11:22:31.445 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:31.466+0000 7f666764e640 10 monclient: init 2024-03-29T11:22:31.445 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:31.466+0000 7f666764e640 10 monclient: _reopen_session rank -1 2024-03-29T11:22:31.445 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:31.466+0000 7f666764e640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:22:31.445 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:31.466+0000 7f666764e640 10 monclient(hunting): picked mon.noname-c con 0x7f66600b39f0 addr v1:172.21.15.84:6790/0 2024-03-29T11:22:31.445 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:31.466+0000 7f666764e640 10 monclient(hunting): picked mon.noname-a con 0x7f66600b4790 addr v1:172.21.15.84:6789/0 2024-03-29T11:22:31.446 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:31.469+0000 7f666764e640 10 monclient(hunting): picked mon.noname-b con 0x7f66601e4e60 addr v1:172.21.15.179:6789/0 2024-03-29T11:22:31.446 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:31.469+0000 7f6665beb640 1 --1- >> v1:172.21.15.179:6789/0 conn(0x7f66601e4e60 0x7f66601eed00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.179:6789/0 says I am v1:172.21.15.84:35073/0 (socket says 172.21.15.84:35073) 2024-03-29T11:22:31.446 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:31.469+0000 7f6665beb640 1 -- 172.21.15.84:0/1293456010 learned_addr learned my addr 172.21.15.84:0/1293456010 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-29T11:22:31.446 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:31.469+0000 7f666764e640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f66601e42c0 con 0x7f66600b4790 2024-03-29T11:22:31.447 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:31.469+0000 7f666764e640 1 -- 172.21.15.84:0/1293456010 --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f66600ac880 con 0x7f66601e4e60 2024-03-29T11:22:31.447 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:31.469+0000 7f666764e640 1 -- 172.21.15.84:0/1293456010 --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f666014e5b0 con 0x7f66600b39f0 2024-03-29T11:22:31.447 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:31.469+0000 7f666764e640 10 monclient(hunting): _renew_subs 2024-03-29T11:22:31.447 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:31.469+0000 7f666764e640 10 monclient(hunting): authenticate will time out at 1012.763868s 2024-03-29T11:22:31.453 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:31.472+0000 7f6655ffb640 10 client.?.objecter ms_handle_connect 0x7f66601e4e60 2024-03-29T11:22:31.453 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:31.472+0000 7f6655ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:22:31.453 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:31.472+0000 7f6655ffb640 1 -- 172.21.15.84:0/1293456010 <== mon.1 v1:172.21.15.179:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3031824041 0 0) 0x7f66600ac880 con 0x7f66601e4e60 2024-03-29T11:22:31.453 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:31.472+0000 7f6655ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:22:31.453 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:31.472+0000 7f6655ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:22:31.453 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:31.472+0000 7f6655ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:22:31.454 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:31.472+0000 7f6655ffb640 10 monclient(hunting): my global_id is 4312 2024-03-29T11:22:31.454 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:31.472+0000 7f6655ffb640 1 -- 172.21.15.84:0/1293456010 --> v1:172.21.15.179:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6644004eb0 con 0x7f66601e4e60 2024-03-29T11:22:31.454 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:31.472+0000 7f6655ffb640 10 client.?.objecter ms_handle_connect 0x7f66600b39f0 2024-03-29T11:22:31.454 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:31.472+0000 7f6655ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:22:31.454 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:31.473+0000 7f6655ffb640 1 -- 172.21.15.84:0/1293456010 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 728771999 0 0) 0x7f666014e5b0 con 0x7f66600b39f0 2024-03-29T11:22:31.454 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:31.473+0000 7f6655ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:22:31.454 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:31.473+0000 7f6655ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:22:31.454 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:31.473+0000 7f6655ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:22:31.454 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:31.473+0000 7f6655ffb640 10 monclient(hunting): my global_id is 4343 2024-03-29T11:22:31.454 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:31.473+0000 7f6655ffb640 1 -- 172.21.15.84:0/1293456010 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f66600ac880 con 0x7f66600b39f0 2024-03-29T11:22:31.454 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:31.473+0000 7f6655ffb640 1 -- 172.21.15.84:0/1293456010 <== mon.1 v1:172.21.15.179:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1436927085 0 0) 0x7f6644004eb0 con 0x7f66601e4e60 2024-03-29T11:22:31.454 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:31.473+0000 7f6655ffb640 1 -- 172.21.15.84:0/1293456010 --> v1:172.21.15.179:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f666014e5b0 con 0x7f66601e4e60 2024-03-29T11:22:31.454 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:31.473+0000 7f6655ffb640 1 -- 172.21.15.84:0/1293456010 <== mon.1 v1:172.21.15.179:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4236868694 0 0) 0x7f665c001af0 con 0x7f66601e4e60 2024-03-29T11:22:31.454 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:31.473+0000 7f6655ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:31.455 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:31.473+0000 7f6655ffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:22:31.455 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:31.473+0000 7f6655ffb640 10 monclient(hunting): dump: 2024-03-29T11:22:31.455 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T11:22:31.455 INFO:teuthology.misc.health.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:22:31.455 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T11:18:53.864025+0000 2024-03-29T11:22:31.455 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T11:18:53.864025+0000 2024-03-29T11:22:31.455 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:31.455 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:22:31.455 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T11:22:31.455 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T11:22:31.455 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T11:22:31.455 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:22:31.455 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:31.473+0000 7f6655ffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:22:31.455 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:31.473+0000 7f6655ffb640 1 -- 172.21.15.84:0/1293456010 <== mon.2 v1:172.21.15.84:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 490244275 0 0) 0x7f66600ac880 con 0x7f66600b39f0 2024-03-29T11:22:31.455 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:31.473+0000 7f6655ffb640 1 -- 172.21.15.84:0/1293456010 --> v1:172.21.15.84:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6644004eb0 con 0x7f66600b39f0 2024-03-29T11:22:31.455 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:31.473+0000 7f6655ffb640 1 -- 172.21.15.84:0/1293456010 <== mon.1 v1:172.21.15.179:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3084854653 0 0) 0x7f666014e5b0 con 0x7f66601e4e60 2024-03-29T11:22:31.455 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:31.473+0000 7f6655ffb640 1 -- 172.21.15.84:0/1293456010 >> v1:172.21.15.84:6790/0 conn(0x7f66600b39f0 legacy=0x7f66600aad20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:31.456 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:31.473+0000 7f6655ffb640 1 -- 172.21.15.84:0/1293456010 >> v1:172.21.15.84:6789/0 conn(0x7f66600b4790 legacy=0x7f66601e4750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:31.456 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:31.473+0000 7f6655ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:22:31.456 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:31.473+0000 7f6655ffb640 1 monclient: found mon.b 2024-03-29T11:22:31.456 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:31.473+0000 7f6655ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:22:31.456 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:31.473+0000 7f6655ffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-29T11:22:31.456 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:31.473+0000 7f6655ffb640 1 -- 172.21.15.84:0/1293456010 --> v1:172.21.15.179:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66600703c0 con 0x7f66601e4e60 2024-03-29T11:22:31.456 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:31.473+0000 7f6655ffb640 10 monclient: _finish_auth 0 2024-03-29T11:22:31.456 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:31.473+0000 7f6655ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:22:31.456 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:31.473+0000 7f6655ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:22:31.456 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:31.473+0000 7f6655ffb640 10 client.?.objecter ms_handle_connect 0x7f66600b4790 2024-03-29T11:22:31.456 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:31.473+0000 7f6655ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:22:31.456 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:31.473+0000 7f666764e640 5 monclient: authenticate success, global_id 4312 2024-03-29T11:22:31.456 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:31.473+0000 7f666764e640 10 monclient: _renew_subs 2024-03-29T11:22:31.456 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:31.473+0000 7f666764e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-29T11:22:31.456 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:31.473+0000 7f666764e640 1 -- 172.21.15.84:0/1293456010 --> v1:172.21.15.179:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f66600ab430 con 0x7f66601e4e60 2024-03-29T11:22:31.456 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:31.474+0000 7f666764e640 10 client.4312.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:22:31.456 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:31.474+0000 7f666764e640 10 monclient: _renew_subs 2024-03-29T11:22:31.457 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:31.474+0000 7f666764e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-29T11:22:31.457 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:31.474+0000 7f666764e640 1 -- 172.21.15.84:0/1293456010 --> v1:172.21.15.179:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f66601f0420 con 0x7f66601e4e60 2024-03-29T11:22:31.457 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:31.474+0000 7f666764e640 1 librados: init done 2024-03-29T11:22:31.457 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:31.475+0000 7f666764e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:22:31.458 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:31.479+0000 7f666764e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:22:31.458 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:31.479+0000 7f666764e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-29T11:22:31.458 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:31.479+0000 7f6655ffb640 1 -- 172.21.15.84:0/1293456010 <== mon.1 v1:172.21.15.179:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f665c001c50 con 0x7f66601e4e60 2024-03-29T11:22:31.458 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:31.479+0000 7f666764e640 1 -- 172.21.15.84:0/1293456010 --> v1:172.21.15.179:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f66600afdc0 con 0x7f66601e4e60 2024-03-29T11:22:31.458 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:31.479+0000 7f6655ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:22:31.458 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:31.479+0000 7f6655ffb640 1 -- 172.21.15.84:0/1293456010 <== mon.1 v1:172.21.15.179:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4236868694 0 0) 0x7f665c004c30 con 0x7f66601e4e60 2024-03-29T11:22:31.458 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:31.479+0000 7f6655ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:31.458 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:31.479+0000 7f6655ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:22:31.458 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:31.479+0000 7f6655ffb640 10 monclient: dump: 2024-03-29T11:22:31.458 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T11:22:31.458 INFO:teuthology.misc.health.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:22:31.458 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T11:18:53.864025+0000 2024-03-29T11:22:31.458 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T11:18:53.864025+0000 2024-03-29T11:22:31.458 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:31.459 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:22:31.459 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T11:22:31.459 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T11:22:31.459 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T11:22:31.459 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:22:31.459 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:31.479+0000 7f66577fe640 4 set_mon_vals no callback set 2024-03-29T11:22:31.459 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:31.480+0000 7f6655ffb640 1 -- 172.21.15.84:0/1293456010 <== mon.1 v1:172.21.15.179:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4033942909 0 0) 0x7f665c01ac00 con 0x7f66601e4e60 2024-03-29T11:22:31.460 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:31.481+0000 7f6655ffb640 1 -- 172.21.15.84:0/1293456010 <== mon.1 v1:172.21.15.179:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2808139587 0 0) 0x7f665c084a70 con 0x7f66601e4e60 2024-03-29T11:22:31.461 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:31.481+0000 7f6655ffb640 10 client.4312.objecter ms_dispatch 0x7f6660000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:22:31.461 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:31.481+0000 7f6655ffb640 3 client.4312.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:22:31.461 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:31.481+0000 7f6655ffb640 3 client.4312.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:22:31.461 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:31.481+0000 7f6655ffb640 20 client.4312.objecter dump_active .. 0 homeless 2024-03-29T11:22:31.461 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:31.484+0000 7f66653ea640 10 monclient: get_auth_request con 0x7f664406dcf0 auth_method 0 2024-03-29T11:22:31.465 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:31.489+0000 7f6655ffb640 10 client.4312.objecter ms_handle_connect 0x7f664406dcf0 2024-03-29T11:22:31.465 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:31.489+0000 7f6655ffb640 1 -- 172.21.15.84:0/1293456010 <== mon.1 v1:172.21.15.179:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f665c050410 con 0x7f66601e4e60 2024-03-29T11:22:31.465 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:31.489+0000 7f6655ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:22:31.466 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:31.489+0000 7f6655ffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:22:31.788 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:31.812+0000 7f666764e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:22:31.788 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:31.812+0000 7f666764e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:22:31.788 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:31.812+0000 7f666764e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-29T11:22:31.789 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:31.812+0000 7f666764e640 1 -- 172.21.15.84:0/1293456010 --> v1:172.21.15.179:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f66600b37b0 con 0x7f66601e4e60 2024-03-29T11:22:31.789 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:31.813+0000 7f6655ffb640 1 -- 172.21.15.84:0/1293456010 <== mon.1 v1:172.21.15.179:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4209477549) 0x7f665c052340 con 0x7f66601e4e60 2024-03-29T11:22:31.790 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:31.813+0000 7f6655ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:22:31.790 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:31.813+0000 7f6655ffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:22:31.790 INFO:teuthology.misc.health.smithi084.stdout: 2024-03-29T11:22:31.790 INFO:teuthology.misc.health.smithi084.stdout:{"epoch":10,"fsid":"2c3e002f-9eb0-4860-853b-4ef1ef562213","created":"2024-03-29T11:19:32.029169+0000","modified":"2024-03-29T11:19:34.200299+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:34.200299+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"5c39e276-2034-4716-bbc0-57a634eefa92","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"cd580e99-109f-4bf1-88c0-a5987bc8b9ed","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"11e60a5a-051d-453f-a190-66387cb35592","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"2c329fbe-bbc4-4030-bf87-d711cad04679","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f4213d4a-bd36-4433-91ae-d7e2b02481d3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"f0cf44ea-6055-48af-805b-aed858102ff9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"d895eaa0-cc0b-4ee4-9b5a-2dfcd89a60b7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"9bc5a7bc-8f52-4db9-96d6-de5e68d52eaa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:22:31.794 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:31.817+0000 7f666764e640 10 librados: watch_flush enter 2024-03-29T11:22:31.794 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:31.817+0000 7f666764e640 10 librados: watch_flush exit 2024-03-29T11:22:31.794 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:31.817+0000 7f666764e640 20 client.4312.objecter shutdown clearing up homeless session... 2024-03-29T11:22:31.794 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:31.817+0000 7f666764e640 10 client.4312.objecter successfully canceled tick 2024-03-29T11:22:31.794 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:31.817+0000 7f666764e640 1 -- 172.21.15.84:0/1293456010 >> v1:172.21.15.179:6816/2981134980 conn(0x7f664406dcf0 legacy=0x7f66440701b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:31.794 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:31.817+0000 7f666764e640 10 monclient: shutdown 2024-03-29T11:22:31.794 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:31.818+0000 7f666764e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:22:31.794 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:31.818+0000 7f666764e640 1 -- 172.21.15.84:0/1293456010 >> v1:172.21.15.179:6789/0 conn(0x7f66601e4e60 legacy=0x7f66601eed00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:31.794 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:31.818+0000 7f666764e640 1 -- 172.21.15.84:0/1293456010 shutdown_connections 2024-03-29T11:22:31.794 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:31.818+0000 7f666764e640 1 -- 172.21.15.84:0/1293456010 >> 172.21.15.84:0/1293456010 conn(0x7f6660066a00 msgr2=0x7f6660066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:22:31.795 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:31.818+0000 7f666764e640 1 -- 172.21.15.84:0/1293456010 shutdown_connections 2024-03-29T11:22:31.795 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:31.818+0000 7f666764e640 1 -- 172.21.15.84:0/1293456010 wait complete. 2024-03-29T11:22:31.795 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:31.818+0000 7f666764e640 1 librados: shutdown 2024-03-29T11:22:31.811 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:22:38.618 DEBUG:teuthology.orchestra.run.smithi084:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:22:38.776 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:38.798+0000 7fa26007b640 10 monclient: get_monmap_and_config 2024-03-29T11:22:38.776 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:38.798+0000 7fa26007b640 10 monclient: build_initial_monmap 2024-03-29T11:22:38.776 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:38.798+0000 7fa26007b640 10 monclient: monmap: 2024-03-29T11:22:38.776 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-29T11:22:38.776 INFO:teuthology.misc.health.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:22:38.776 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T11:22:38.799999+0000 2024-03-29T11:22:38.776 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T11:22:38.799999+0000 2024-03-29T11:22:38.776 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-29T11:22:38.776 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:22:38.777 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-29T11:22:38.777 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.noname-c 2024-03-29T11:22:38.777 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-b 2024-03-29T11:22:38.777 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:22:38.777 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:38.798+0000 7fa26007b640 1 Processor -- start 2024-03-29T11:22:38.778 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:38.799+0000 7fa26007b640 1 -- start start 2024-03-29T11:22:38.778 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:38.799+0000 7fa26007b640 10 monclient: init 2024-03-29T11:22:38.778 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:38.800+0000 7fa26007b640 10 monclient: _reopen_session rank -1 2024-03-29T11:22:38.778 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:38.800+0000 7fa26007b640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:22:38.778 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:38.800+0000 7fa26007b640 10 monclient(hunting): picked mon.noname-a con 0x7fa2580b0770 addr v1:172.21.15.84:6789/0 2024-03-29T11:22:38.778 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:38.800+0000 7fa26007b640 10 monclient(hunting): picked mon.noname-b con 0x7fa25814a700 addr v1:172.21.15.84:6790/0 2024-03-29T11:22:38.778 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:38.800+0000 7fa26007b640 10 monclient(hunting): picked mon.noname-c con 0x7fa2580ac5a0 addr v1:172.21.15.179:6789/0 2024-03-29T11:22:38.778 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:38.800+0000 7fa26007b640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa2580aba70 con 0x7fa2580b0770 2024-03-29T11:22:38.778 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:38.800+0000 7fa26007b640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa2580aecb0 con 0x7fa2580ac5a0 2024-03-29T11:22:38.778 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:38.800+0000 7fa26007b640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa2580af150 con 0x7fa25814a700 2024-03-29T11:22:38.778 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:38.800+0000 7fa26007b640 10 monclient(hunting): _renew_subs 2024-03-29T11:22:38.779 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:38.800+0000 7fa26007b640 10 monclient(hunting): authenticate will time out at 1020.095210s 2024-03-29T11:22:38.781 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:38.803+0000 7fa25e618640 1 --1- >> v1:172.21.15.179:6789/0 conn(0x7fa2580ac5a0 0x7fa258151f00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.179:6789/0 says I am v1:172.21.15.84:39581/0 (socket says 172.21.15.84:39581) 2024-03-29T11:22:38.781 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:38.803+0000 7fa25e618640 1 -- 172.21.15.84:0/3164926954 learned_addr learned my addr 172.21.15.84:0/3164926954 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-29T11:22:38.784 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:38.806+0000 7fa25ce15640 1 -- 172.21.15.84:0/3164926954 <== mon.1 v1:172.21.15.179:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 928018530 0 0) 0x7fa2580aecb0 con 0x7fa2580ac5a0 2024-03-29T11:22:38.784 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:38.806+0000 7fa25ce15640 10 monclient(hunting): _negotiate 2024-03-29T11:22:38.785 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:38.806+0000 7fa25ce15640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:22:38.785 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:38.806+0000 7fa25ce15640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:22:38.785 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:38.806+0000 7fa25ce15640 10 monclient(hunting): my global_id is 4315 2024-03-29T11:22:38.785 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:38.806+0000 7fa25ce15640 1 -- 172.21.15.84:0/3164926954 --> v1:172.21.15.179:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa244003520 con 0x7fa2580ac5a0 2024-03-29T11:22:38.785 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:38.807+0000 7fa25ce15640 1 -- 172.21.15.84:0/3164926954 <== mon.1 v1:172.21.15.179:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1979196763 0 0) 0x7fa244003520 con 0x7fa2580ac5a0 2024-03-29T11:22:38.785 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:38.807+0000 7fa25ce15640 1 -- 172.21.15.84:0/3164926954 >> v1:172.21.15.84:6790/0 conn(0x7fa25814a700 legacy=0x7fa2580abe90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:22:38.785 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:38.807+0000 7fa25ce15640 1 -- 172.21.15.84:0/3164926954 >> v1:172.21.15.84:6789/0 conn(0x7fa2580b0770 legacy=0x7fa258149ff0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:22:38.787 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:38.809+0000 7fa25ce15640 10 monclient: _finish_hunting 0 2024-03-29T11:22:38.787 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:38.809+0000 7fa25ce15640 1 monclient: found mon.noname-c 2024-03-29T11:22:38.787 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:38.809+0000 7fa25ce15640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:22:38.787 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:38.809+0000 7fa25ce15640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.179:6789/0 2024-03-29T11:22:38.787 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:38.809+0000 7fa25ce15640 1 -- 172.21.15.84:0/3164926954 --> v1:172.21.15.179:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2580ac9d0 con 0x7fa2580ac5a0 2024-03-29T11:22:38.788 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:38.810+0000 7fa25ce15640 10 monclient: _finish_auth 0 2024-03-29T11:22:38.788 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:38.810+0000 7fa25ce15640 10 monclient: _check_auth_tickets 2024-03-29T11:22:38.788 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:38.810+0000 7fa25ce15640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:22:38.788 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:38.810+0000 7fa26007b640 5 monclient: authenticate success, global_id 4315 2024-03-29T11:22:38.788 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:38.810+0000 7fa26007b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:22:38.788 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:38.810+0000 7fa25ce15640 1 -- 172.21.15.84:0/3164926954 <== mon.1 v1:172.21.15.179:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4236868694 0 0) 0x7fa250001a30 con 0x7fa2580ac5a0 2024-03-29T11:22:38.788 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:38.810+0000 7fa25ce15640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:38.788 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:38.810+0000 7fa25ce15640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:22:38.788 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:38.810+0000 7fa25ce15640 10 monclient: dump: 2024-03-29T11:22:38.788 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T11:22:38.788 INFO:teuthology.misc.health.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:22:38.788 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T11:18:53.864025+0000 2024-03-29T11:22:38.788 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T11:18:53.864025+0000 2024-03-29T11:22:38.788 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:38.788 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:22:38.789 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T11:22:38.789 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T11:22:38.789 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T11:22:38.789 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:22:38.789 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:38.810+0000 7fa26007b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:22:38.789 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:38.810+0000 7fa25ce15640 1 -- 172.21.15.84:0/3164926954 <== mon.1 v1:172.21.15.179:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa250001cc0 con 0x7fa2580ac5a0 2024-03-29T11:22:38.789 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:38.810+0000 7fa25ce15640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:22:38.789 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:38.810+0000 7fa26007b640 10 monclient: get_monmap_and_config success 2024-03-29T11:22:38.790 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:38.810+0000 7fa26007b640 4 set_mon_vals no callback set 2024-03-29T11:22:38.790 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:38.811+0000 7fa26007b640 10 monclient: shutdown 2024-03-29T11:22:38.790 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:38.811+0000 7fa26007b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:22:38.790 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:38.811+0000 7fa26007b640 1 -- 172.21.15.84:0/3164926954 >> v1:172.21.15.179:6789/0 conn(0x7fa2580ac5a0 legacy=0x7fa258151f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:38.790 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:38.811+0000 7fa26007b640 1 -- 172.21.15.84:0/3164926954 shutdown_connections 2024-03-29T11:22:38.790 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:38.811+0000 7fa26007b640 1 -- 172.21.15.84:0/3164926954 >> 172.21.15.84:0/3164926954 conn(0x7fa25806b660 msgr2=0x7fa25806ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:22:38.791 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:38.811+0000 7fa26007b640 1 -- 172.21.15.84:0/3164926954 shutdown_connections 2024-03-29T11:22:38.791 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:38.811+0000 7fa26007b640 1 -- 172.21.15.84:0/3164926954 wait complete. 2024-03-29T11:22:38.791 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:38.812+0000 7fa26007b640 10 monclient: build_initial_monmap 2024-03-29T11:22:38.792 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:38.812+0000 7fa26007b640 10 monclient: monmap: 2024-03-29T11:22:38.792 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-29T11:22:38.792 INFO:teuthology.misc.health.smithi084.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:22:38.792 INFO:teuthology.misc.health.smithi084.stderr:last_changed 0.000000 2024-03-29T11:22:38.792 INFO:teuthology.misc.health.smithi084.stderr:created 0.000000 2024-03-29T11:22:38.792 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-29T11:22:38.792 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:22:38.792 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-29T11:22:38.792 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.noname-b 2024-03-29T11:22:38.792 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-c 2024-03-29T11:22:38.792 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:22:38.792 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:38.812+0000 7fa26007b640 1 librados: starting msgr at 2024-03-29T11:22:38.792 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:38.812+0000 7fa26007b640 1 librados: starting objecter 2024-03-29T11:22:38.792 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:38.812+0000 7fa26007b640 1 Processor -- start 2024-03-29T11:22:38.792 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:38.813+0000 7fa26007b640 1 -- start start 2024-03-29T11:22:38.793 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:38.813+0000 7fa26007b640 1 librados: setting wanted keys 2024-03-29T11:22:38.793 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:38.813+0000 7fa26007b640 1 librados: calling monclient init 2024-03-29T11:22:38.793 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:38.813+0000 7fa26007b640 10 monclient: init 2024-03-29T11:22:38.793 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:38.813+0000 7fa26007b640 10 monclient: _reopen_session rank -1 2024-03-29T11:22:38.793 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:38.813+0000 7fa26007b640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:22:38.793 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:38.813+0000 7fa26007b640 10 monclient(hunting): picked mon.noname-a con 0x7fa2580ac5a0 addr v1:172.21.15.84:6789/0 2024-03-29T11:22:38.793 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:38.813+0000 7fa26007b640 10 monclient(hunting): picked mon.noname-b con 0x7fa2580b0770 addr v1:172.21.15.179:6789/0 2024-03-29T11:22:38.793 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:38.813+0000 7fa26007b640 10 monclient(hunting): picked mon.noname-c con 0x7fa25814a700 addr v1:172.21.15.84:6790/0 2024-03-29T11:22:38.793 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:38.813+0000 7fa26007b640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa25807f4a0 con 0x7fa2580ac5a0 2024-03-29T11:22:38.793 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:38.813+0000 7fa26007b640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa25807de50 con 0x7fa2580b0770 2024-03-29T11:22:38.793 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:38.813+0000 7fa26007b640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa2581474d0 con 0x7fa25814a700 2024-03-29T11:22:38.793 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:38.813+0000 7fa26007b640 10 monclient(hunting): _renew_subs 2024-03-29T11:22:38.793 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:38.813+0000 7fa26007b640 10 monclient(hunting): authenticate will time out at 1020.109035s 2024-03-29T11:22:38.797 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:38.818+0000 7fa25d616640 1 --1- >> v1:172.21.15.179:6789/0 conn(0x7fa2580b0770 0x7fa2581e9490 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.179:6789/0 says I am v1:172.21.15.84:39675/0 (socket says 172.21.15.84:39675) 2024-03-29T11:22:38.798 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:38.818+0000 7fa25d616640 1 -- 172.21.15.84:0/477687554 learned_addr learned my addr 172.21.15.84:0/477687554 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-29T11:22:38.801 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:38.818+0000 7fa24a7fc640 10 client.?.objecter ms_handle_connect 0x7fa2580b0770 2024-03-29T11:22:38.801 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:38.818+0000 7fa24a7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:22:38.801 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:38.820+0000 7fa24a7fc640 10 client.?.objecter ms_handle_connect 0x7fa2580ac5a0 2024-03-29T11:22:38.801 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:38.820+0000 7fa24a7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:22:38.802 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:38.820+0000 7fa24a7fc640 1 -- 172.21.15.84:0/477687554 <== mon.1 v1:172.21.15.179:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 555418900 0 0) 0x7fa25807de50 con 0x7fa2580b0770 2024-03-29T11:22:38.802 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:38.820+0000 7fa24a7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:22:38.802 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:38.820+0000 7fa24a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:22:38.802 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:38.820+0000 7fa24a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:22:38.802 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:38.820+0000 7fa24a7fc640 10 monclient(hunting): my global_id is 4318 2024-03-29T11:22:38.802 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:38.820+0000 7fa24a7fc640 1 -- 172.21.15.84:0/477687554 --> v1:172.21.15.179:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa23c004eb0 con 0x7fa2580b0770 2024-03-29T11:22:38.802 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:38.821+0000 7fa24a7fc640 1 -- 172.21.15.84:0/477687554 <== mon.0 v1:172.21.15.84:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3332260075 0 0) 0x7fa25807f4a0 con 0x7fa2580ac5a0 2024-03-29T11:22:38.802 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:38.821+0000 7fa24a7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:22:38.802 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:38.821+0000 7fa24a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:22:38.802 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:38.821+0000 7fa24a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:22:38.802 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:38.821+0000 7fa24a7fc640 10 monclient(hunting): my global_id is 4353 2024-03-29T11:22:38.802 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:38.821+0000 7fa24a7fc640 1 -- 172.21.15.84:0/477687554 --> v1:172.21.15.84:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa25807de50 con 0x7fa2580ac5a0 2024-03-29T11:22:38.802 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:38.822+0000 7fa24a7fc640 1 -- 172.21.15.84:0/477687554 <== mon.0 v1:172.21.15.84:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2635543221 0 0) 0x7fa25807de50 con 0x7fa2580ac5a0 2024-03-29T11:22:38.802 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:38.822+0000 7fa24a7fc640 1 -- 172.21.15.84:0/477687554 --> v1:172.21.15.84:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa25807f4a0 con 0x7fa2580ac5a0 2024-03-29T11:22:38.803 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:38.822+0000 7fa24a7fc640 1 -- 172.21.15.84:0/477687554 <== mon.0 v1:172.21.15.84:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4236868694 0 0) 0x7fa24c001830 con 0x7fa2580ac5a0 2024-03-29T11:22:38.803 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:38.822+0000 7fa24a7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:38.803 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:38.822+0000 7fa24a7fc640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:22:38.803 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:38.822+0000 7fa24a7fc640 10 monclient(hunting): dump: 2024-03-29T11:22:38.803 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T11:22:38.803 INFO:teuthology.misc.health.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:22:38.803 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T11:18:53.864025+0000 2024-03-29T11:22:38.803 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T11:18:53.864025+0000 2024-03-29T11:22:38.803 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:38.803 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:22:38.803 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T11:22:38.803 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T11:22:38.803 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T11:22:38.803 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:22:38.803 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:38.822+0000 7fa24a7fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:22:38.804 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:38.822+0000 7fa24a7fc640 1 -- 172.21.15.84:0/477687554 <== mon.1 v1:172.21.15.179:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3699388343 0 0) 0x7fa23c004eb0 con 0x7fa2580b0770 2024-03-29T11:22:38.804 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:38.822+0000 7fa24a7fc640 1 -- 172.21.15.84:0/477687554 --> v1:172.21.15.179:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa25807de50 con 0x7fa2580b0770 2024-03-29T11:22:38.804 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:38.822+0000 7fa24a7fc640 1 -- 172.21.15.84:0/477687554 <== mon.1 v1:172.21.15.179:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4236868694 0 0) 0x7fa254003fe0 con 0x7fa2580b0770 2024-03-29T11:22:38.804 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:38.822+0000 7fa24a7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:38.804 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:38.822+0000 7fa24a7fc640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:22:38.804 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:38.822+0000 7fa24a7fc640 10 monclient(hunting): dump: 2024-03-29T11:22:38.804 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T11:22:38.804 INFO:teuthology.misc.health.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:22:38.804 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T11:18:53.864025+0000 2024-03-29T11:22:38.804 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T11:18:53.864025+0000 2024-03-29T11:22:38.804 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:38.804 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:22:38.804 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T11:22:38.804 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T11:22:38.805 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T11:22:38.805 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:22:38.805 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:38.822+0000 7fa24a7fc640 10 client.?.objecter ms_handle_connect 0x7fa25814a700 2024-03-29T11:22:38.805 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:38.822+0000 7fa24a7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:22:38.805 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:38.822+0000 7fa24a7fc640 1 -- 172.21.15.84:0/477687554 <== mon.1 v1:172.21.15.179:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1578979418 0 0) 0x7fa25807de50 con 0x7fa2580b0770 2024-03-29T11:22:38.805 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:38.822+0000 7fa24a7fc640 1 -- 172.21.15.84:0/477687554 >> v1:172.21.15.84:6790/0 conn(0x7fa25814a700 legacy=0x7fa2581ec3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:38.805 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:38.822+0000 7fa24a7fc640 1 -- 172.21.15.84:0/477687554 >> v1:172.21.15.84:6789/0 conn(0x7fa2580ac5a0 legacy=0x7fa25807d540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:38.805 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:38.822+0000 7fa24a7fc640 10 monclient: _finish_hunting 0 2024-03-29T11:22:38.805 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:38.822+0000 7fa24a7fc640 1 monclient: found mon.b 2024-03-29T11:22:38.805 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:38.822+0000 7fa24a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:22:38.805 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:38.822+0000 7fa24a7fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-29T11:22:38.805 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:38.822+0000 7fa24a7fc640 1 -- 172.21.15.84:0/477687554 --> v1:172.21.15.179:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa258147660 con 0x7fa2580b0770 2024-03-29T11:22:38.805 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:38.822+0000 7fa24a7fc640 10 monclient: _finish_auth 0 2024-03-29T11:22:38.805 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:38.822+0000 7fa24a7fc640 10 monclient: _check_auth_tickets 2024-03-29T11:22:38.805 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:38.822+0000 7fa24a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:22:38.806 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:38.822+0000 7fa26007b640 5 monclient: authenticate success, global_id 4318 2024-03-29T11:22:38.806 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:38.822+0000 7fa26007b640 10 monclient: _renew_subs 2024-03-29T11:22:38.806 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:38.822+0000 7fa26007b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-29T11:22:38.806 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:38.822+0000 7fa26007b640 1 -- 172.21.15.84:0/477687554 --> v1:172.21.15.179:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa2581e9ba0 con 0x7fa2580b0770 2024-03-29T11:22:38.806 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:38.822+0000 7fa26007b640 10 client.4318.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:22:38.806 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:38.822+0000 7fa26007b640 10 monclient: _renew_subs 2024-03-29T11:22:38.806 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:38.822+0000 7fa26007b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-29T11:22:38.806 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:38.822+0000 7fa26007b640 1 -- 172.21.15.84:0/477687554 --> v1:172.21.15.179:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa2581ecad0 con 0x7fa2580b0770 2024-03-29T11:22:38.806 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:38.822+0000 7fa26007b640 1 librados: init done 2024-03-29T11:22:38.806 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:38.823+0000 7fa24a7fc640 1 -- 172.21.15.84:0/477687554 <== mon.1 v1:172.21.15.179:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa254000ed0 con 0x7fa2580b0770 2024-03-29T11:22:38.806 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:38.823+0000 7fa24a7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:22:38.806 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:38.823+0000 7fa24a7fc640 1 -- 172.21.15.84:0/477687554 <== mon.1 v1:172.21.15.179:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4236868694 0 0) 0x7fa2540012a0 con 0x7fa2580b0770 2024-03-29T11:22:38.806 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:38.823+0000 7fa24a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:38.806 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:38.823+0000 7fa24a7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:22:38.807 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:38.823+0000 7fa24a7fc640 10 monclient: dump: 2024-03-29T11:22:38.807 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T11:22:38.807 INFO:teuthology.misc.health.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:22:38.807 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T11:18:53.864025+0000 2024-03-29T11:22:38.807 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T11:18:53.864025+0000 2024-03-29T11:22:38.807 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:38.807 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:22:38.807 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T11:22:38.807 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T11:22:38.807 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T11:22:38.807 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:22:38.807 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:38.823+0000 7fa24bfff640 4 set_mon_vals no callback set 2024-03-29T11:22:38.808 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:38.824+0000 7fa26007b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:22:38.808 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:38.824+0000 7fa26007b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:22:38.808 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:38.824+0000 7fa26007b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-29T11:22:38.808 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:38.824+0000 7fa26007b640 1 -- 172.21.15.84:0/477687554 --> v1:172.21.15.179:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa240005080 con 0x7fa2580b0770 2024-03-29T11:22:38.808 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:38.825+0000 7fa24a7fc640 1 -- 172.21.15.84:0/477687554 <== mon.1 v1:172.21.15.179:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4033942909 0 0) 0x7fa254001400 con 0x7fa2580b0770 2024-03-29T11:22:38.808 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:38.826+0000 7fa24a7fc640 1 -- 172.21.15.84:0/477687554 <== mon.1 v1:172.21.15.179:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2808139587 0 0) 0x7fa2540547b0 con 0x7fa2580b0770 2024-03-29T11:22:38.808 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:38.826+0000 7fa24a7fc640 10 client.4318.objecter ms_dispatch 0x7fa258000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:22:38.808 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:38.826+0000 7fa24a7fc640 3 client.4318.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:22:38.809 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:38.826+0000 7fa24a7fc640 3 client.4318.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:22:38.809 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:38.826+0000 7fa24a7fc640 20 client.4318.objecter dump_active .. 0 homeless 2024-03-29T11:22:38.809 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:38.826+0000 7fa25de17640 10 monclient: get_auth_request con 0x7fa23c06de70 auth_method 0 2024-03-29T11:22:38.809 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:38.828+0000 7fa24a7fc640 10 client.4318.objecter ms_handle_connect 0x7fa23c06de70 2024-03-29T11:22:38.812 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:38.834+0000 7fa24a7fc640 1 -- 172.21.15.84:0/477687554 <== mon.1 v1:172.21.15.179:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fa254050a50 con 0x7fa2580b0770 2024-03-29T11:22:38.812 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:38.834+0000 7fa24a7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:22:38.812 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:38.834+0000 7fa24a7fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:22:39.132 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:39.154+0000 7fa26007b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:22:39.132 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:39.154+0000 7fa26007b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:22:39.132 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:39.154+0000 7fa26007b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-29T11:22:39.132 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:39.154+0000 7fa26007b640 1 -- 172.21.15.84:0/477687554 --> v1:172.21.15.179:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa240005330 con 0x7fa2580b0770 2024-03-29T11:22:39.136 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:39.159+0000 7fa24a7fc640 1 -- 172.21.15.84:0/477687554 <== mon.1 v1:172.21.15.179:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4209477549) 0x7fa254050bf0 con 0x7fa2580b0770 2024-03-29T11:22:39.137 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:39.159+0000 7fa24a7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:22:39.137 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:39.159+0000 7fa24a7fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:22:39.140 INFO:teuthology.misc.health.smithi084.stdout: 2024-03-29T11:22:39.140 INFO:teuthology.misc.health.smithi084.stdout:{"epoch":10,"fsid":"2c3e002f-9eb0-4860-853b-4ef1ef562213","created":"2024-03-29T11:19:32.029169+0000","modified":"2024-03-29T11:19:34.200299+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:34.200299+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"5c39e276-2034-4716-bbc0-57a634eefa92","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"cd580e99-109f-4bf1-88c0-a5987bc8b9ed","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"11e60a5a-051d-453f-a190-66387cb35592","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"2c329fbe-bbc4-4030-bf87-d711cad04679","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f4213d4a-bd36-4433-91ae-d7e2b02481d3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"f0cf44ea-6055-48af-805b-aed858102ff9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"d895eaa0-cc0b-4ee4-9b5a-2dfcd89a60b7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"9bc5a7bc-8f52-4db9-96d6-de5e68d52eaa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:22:39.140 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:39.163+0000 7fa26007b640 10 librados: watch_flush enter 2024-03-29T11:22:39.140 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:39.163+0000 7fa26007b640 10 librados: watch_flush exit 2024-03-29T11:22:39.140 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:39.163+0000 7fa26007b640 20 client.4318.objecter shutdown clearing up homeless session... 2024-03-29T11:22:39.140 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:39.163+0000 7fa26007b640 10 client.4318.objecter successfully canceled tick 2024-03-29T11:22:39.144 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:39.166+0000 7fa26007b640 1 -- 172.21.15.84:0/477687554 >> v1:172.21.15.179:6816/2981134980 conn(0x7fa23c06de70 legacy=0x7fa23c070330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:39.144 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:39.166+0000 7fa26007b640 10 monclient: shutdown 2024-03-29T11:22:39.144 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:39.166+0000 7fa26007b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:22:39.144 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:39.166+0000 7fa26007b640 1 -- 172.21.15.84:0/477687554 >> v1:172.21.15.179:6789/0 conn(0x7fa2580b0770 legacy=0x7fa2581e9490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:39.145 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:39.167+0000 7fa26007b640 1 -- 172.21.15.84:0/477687554 shutdown_connections 2024-03-29T11:22:39.145 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:39.167+0000 7fa26007b640 1 -- 172.21.15.84:0/477687554 >> 172.21.15.84:0/477687554 conn(0x7fa25806b660 msgr2=0x7fa258149280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:22:39.145 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:39.167+0000 7fa26007b640 1 -- 172.21.15.84:0/477687554 shutdown_connections 2024-03-29T11:22:39.145 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:39.167+0000 7fa26007b640 1 -- 172.21.15.84:0/477687554 wait complete. 2024-03-29T11:22:39.145 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:39.167+0000 7fa26007b640 1 librados: shutdown 2024-03-29T11:22:39.162 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:22:45.968 DEBUG:teuthology.orchestra.run.smithi084:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:22:46.108 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:46.132+0000 7f3f7540f640 10 monclient: get_monmap_and_config 2024-03-29T11:22:46.109 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:46.132+0000 7f3f7540f640 10 monclient: build_initial_monmap 2024-03-29T11:22:46.109 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:46.132+0000 7f3f7540f640 10 monclient: monmap: 2024-03-29T11:22:46.109 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-29T11:22:46.109 INFO:teuthology.misc.health.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:22:46.109 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T11:22:46.133060+0000 2024-03-29T11:22:46.109 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T11:22:46.133060+0000 2024-03-29T11:22:46.109 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-29T11:22:46.109 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:22:46.109 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-29T11:22:46.109 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.noname-c 2024-03-29T11:22:46.109 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-b 2024-03-29T11:22:46.109 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:22:46.110 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:46.132+0000 7f3f7540f640 1 Processor -- start 2024-03-29T11:22:46.110 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:46.133+0000 7f3f7540f640 1 -- start start 2024-03-29T11:22:46.110 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:46.133+0000 7f3f7540f640 10 monclient: init 2024-03-29T11:22:46.114 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:46.135+0000 7f3f7540f640 10 monclient: _reopen_session rank -1 2024-03-29T11:22:46.114 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:46.135+0000 7f3f7540f640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:22:46.115 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:46.135+0000 7f3f7540f640 10 monclient(hunting): picked mon.noname-c con 0x7f3f70074650 addr v1:172.21.15.179:6789/0 2024-03-29T11:22:46.115 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:46.135+0000 7f3f7540f640 10 monclient(hunting): picked mon.noname-b con 0x7f3f70141ff0 addr v1:172.21.15.84:6790/0 2024-03-29T11:22:46.115 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:46.135+0000 7f3f7540f640 10 monclient(hunting): picked mon.noname-a con 0x7f3f7014b870 addr v1:172.21.15.84:6789/0 2024-03-29T11:22:46.115 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:46.135+0000 7f3f7540f640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3f7006f6f0 con 0x7f3f7014b870 2024-03-29T11:22:46.115 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:46.135+0000 7f3f7540f640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3f7006f9d0 con 0x7f3f70074650 2024-03-29T11:22:46.115 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:46.136+0000 7f3f7540f640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3f7013faa0 con 0x7f3f70141ff0 2024-03-29T11:22:46.115 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:46.136+0000 7f3f6f7fe640 1 --1- >> v1:172.21.15.84:6789/0 conn(0x7f3f7014b870 0x7f3f7014dc60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.84:6789/0 says I am v1:172.21.15.84:55506/0 (socket says 172.21.15.84:55506) 2024-03-29T11:22:46.115 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:46.136+0000 7f3f6f7fe640 1 -- 172.21.15.84:0/3977268068 learned_addr learned my addr 172.21.15.84:0/3977268068 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-29T11:22:46.115 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:46.136+0000 7f3f7540f640 10 monclient(hunting): _renew_subs 2024-03-29T11:22:46.116 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:46.136+0000 7f3f7540f640 10 monclient(hunting): authenticate will time out at 1027.430480s 2024-03-29T11:22:46.116 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:46.136+0000 7f3f6dffb640 1 -- 172.21.15.84:0/3977268068 <== mon.0 v1:172.21.15.84:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1485122670 0 0) 0x7f3f7006f6f0 con 0x7f3f7014b870 2024-03-29T11:22:46.116 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:46.136+0000 7f3f6dffb640 10 monclient(hunting): _negotiate 2024-03-29T11:22:46.116 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:46.136+0000 7f3f6dffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:22:46.116 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:46.136+0000 7f3f6dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:22:46.116 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:46.136+0000 7f3f6dffb640 10 monclient(hunting): my global_id is 4356 2024-03-29T11:22:46.116 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:46.136+0000 7f3f6dffb640 1 -- 172.21.15.84:0/3977268068 --> v1:172.21.15.84:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3f58003520 con 0x7f3f7014b870 2024-03-29T11:22:46.116 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:46.136+0000 7f3f6dffb640 1 -- 172.21.15.84:0/3977268068 <== mon.0 v1:172.21.15.84:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 395740902 0 0) 0x7f3f58003520 con 0x7f3f7014b870 2024-03-29T11:22:46.116 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:46.136+0000 7f3f6dffb640 1 -- 172.21.15.84:0/3977268068 >> v1:172.21.15.84:6790/0 conn(0x7f3f70141ff0 legacy=0x7f3f70142460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:46.116 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:46.137+0000 7f3f6dffb640 1 -- 172.21.15.84:0/3977268068 >> v1:172.21.15.179:6789/0 conn(0x7f3f70074650 legacy=0x7f3f7013bd20 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:22:46.116 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:46.137+0000 7f3f6dffb640 10 monclient: _finish_hunting 0 2024-03-29T11:22:46.116 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:46.137+0000 7f3f6dffb640 1 monclient: found mon.noname-a 2024-03-29T11:22:46.117 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:46.137+0000 7f3f6dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:22:46.117 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:46.137+0000 7f3f6dffb640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.84:6789/0 2024-03-29T11:22:46.117 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:46.137+0000 7f3f6dffb640 1 -- 172.21.15.84:0/3977268068 --> v1:172.21.15.84:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f700aff70 con 0x7f3f7014b870 2024-03-29T11:22:46.117 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:46.137+0000 7f3f6dffb640 10 monclient: _finish_auth 0 2024-03-29T11:22:46.117 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:46.137+0000 7f3f6dffb640 10 monclient: _check_auth_tickets 2024-03-29T11:22:46.117 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:46.137+0000 7f3f6dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:22:46.117 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:46.137+0000 7f3f6dffb640 1 -- 172.21.15.84:0/3977268068 <== mon.0 v1:172.21.15.84:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4236868694 0 0) 0x7f3f68001a10 con 0x7f3f7014b870 2024-03-29T11:22:46.117 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:46.137+0000 7f3f6dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:46.118 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:46.137+0000 7f3f6dffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:22:46.118 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:46.137+0000 7f3f6dffb640 10 monclient: dump: 2024-03-29T11:22:46.118 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T11:22:46.118 INFO:teuthology.misc.health.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:22:46.118 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T11:18:53.864025+0000 2024-03-29T11:22:46.118 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T11:18:53.864025+0000 2024-03-29T11:22:46.118 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:46.118 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:22:46.118 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T11:22:46.118 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T11:22:46.118 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T11:22:46.118 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:22:46.119 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:46.137+0000 7f3f7540f640 5 monclient: authenticate success, global_id 4356 2024-03-29T11:22:46.119 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:46.137+0000 7f3f7540f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:22:46.119 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:46.137+0000 7f3f6dffb640 1 -- 172.21.15.84:0/3977268068 <== mon.0 v1:172.21.15.84:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3f68001ca0 con 0x7f3f7014b870 2024-03-29T11:22:46.121 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:46.137+0000 7f3f6dffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:22:46.121 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:46.137+0000 7f3f6dffb640 1 -- 172.21.15.84:0/3977268068 <== mon.0 v1:172.21.15.84:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4236868694 0 0) 0x7f3f68002030 con 0x7f3f7014b870 2024-03-29T11:22:46.121 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:46.137+0000 7f3f7540f640 10 monclient: get_monmap_and_config success 2024-03-29T11:22:46.121 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:46.137+0000 7f3f7540f640 4 set_mon_vals no callback set 2024-03-29T11:22:46.121 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:46.138+0000 7f3f6dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:46.121 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:46.138+0000 7f3f6dffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:22:46.121 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:46.138+0000 7f3f6dffb640 10 monclient: dump: 2024-03-29T11:22:46.121 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T11:22:46.121 INFO:teuthology.misc.health.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:22:46.121 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T11:18:53.864025+0000 2024-03-29T11:22:46.122 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T11:18:53.864025+0000 2024-03-29T11:22:46.122 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:46.122 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:22:46.122 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T11:22:46.122 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T11:22:46.122 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T11:22:46.122 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:22:46.122 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:46.138+0000 7f3f7540f640 10 monclient: shutdown 2024-03-29T11:22:46.122 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:46.138+0000 7f3f7540f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:22:46.122 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:46.138+0000 7f3f7540f640 1 -- 172.21.15.84:0/3977268068 >> v1:172.21.15.84:6789/0 conn(0x7f3f7014b870 legacy=0x7f3f7014dc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:46.122 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:46.139+0000 7f3f7540f640 1 -- 172.21.15.84:0/3977268068 shutdown_connections 2024-03-29T11:22:46.122 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:46.139+0000 7f3f7540f640 1 -- 172.21.15.84:0/3977268068 >> 172.21.15.84:0/3977268068 conn(0x7f3f700b3f40 msgr2=0x7f3f700b4320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:22:46.123 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:46.140+0000 7f3f7540f640 1 -- 172.21.15.84:0/3977268068 shutdown_connections 2024-03-29T11:22:46.123 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:46.140+0000 7f3f7540f640 1 -- 172.21.15.84:0/3977268068 wait complete. 2024-03-29T11:22:46.123 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:46.140+0000 7f3f7540f640 10 monclient: build_initial_monmap 2024-03-29T11:22:46.123 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:46.140+0000 7f3f7540f640 10 monclient: monmap: 2024-03-29T11:22:46.123 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-29T11:22:46.123 INFO:teuthology.misc.health.smithi084.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:22:46.123 INFO:teuthology.misc.health.smithi084.stderr:last_changed 0.000000 2024-03-29T11:22:46.123 INFO:teuthology.misc.health.smithi084.stderr:created 0.000000 2024-03-29T11:22:46.123 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-29T11:22:46.123 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:22:46.123 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-29T11:22:46.123 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.noname-b 2024-03-29T11:22:46.123 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-c 2024-03-29T11:22:46.124 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:22:46.124 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:46.141+0000 7f3f7540f640 1 librados: starting msgr at 2024-03-29T11:22:46.124 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:46.141+0000 7f3f7540f640 1 librados: starting objecter 2024-03-29T11:22:46.124 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:46.141+0000 7f3f7540f640 1 Processor -- start 2024-03-29T11:22:46.124 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:46.141+0000 7f3f7540f640 1 -- start start 2024-03-29T11:22:46.124 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:46.141+0000 7f3f7540f640 1 librados: setting wanted keys 2024-03-29T11:22:46.124 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:46.141+0000 7f3f7540f640 1 librados: calling monclient init 2024-03-29T11:22:46.124 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:46.141+0000 7f3f7540f640 10 monclient: init 2024-03-29T11:22:46.124 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:46.142+0000 7f3f7540f640 10 monclient: _reopen_session rank -1 2024-03-29T11:22:46.124 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:46.142+0000 7f3f7540f640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:22:46.124 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:46.144+0000 7f3f7540f640 10 monclient(hunting): picked mon.noname-b con 0x7f3f70074650 addr v1:172.21.15.179:6789/0 2024-03-29T11:22:46.124 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:46.144+0000 7f3f7540f640 10 monclient(hunting): picked mon.noname-c con 0x7f3f70141ff0 addr v1:172.21.15.84:6790/0 2024-03-29T11:22:46.124 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:46.144+0000 7f3f7540f640 10 monclient(hunting): picked mon.noname-a con 0x7f3f7014b870 addr v1:172.21.15.84:6789/0 2024-03-29T11:22:46.124 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:46.144+0000 7f3f7540f640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3f70135d50 con 0x7f3f7014b870 2024-03-29T11:22:46.125 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:46.144+0000 7f3f7540f640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3f7013c4f0 con 0x7f3f70074650 2024-03-29T11:22:46.125 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:46.144+0000 7f3f7540f640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3f701d7980 con 0x7f3f70141ff0 2024-03-29T11:22:46.125 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:46.144+0000 7f3f7540f640 10 monclient(hunting): _renew_subs 2024-03-29T11:22:46.125 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:46.144+0000 7f3f7540f640 10 monclient(hunting): authenticate will time out at 1027.438237s 2024-03-29T11:22:46.130 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:46.145+0000 7f3f6effd640 1 --1- >> v1:172.21.15.179:6789/0 conn(0x7f3f70074650 0x7f3f701d7270 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.179:6789/0 says I am v1:172.21.15.84:54479/0 (socket says 172.21.15.84:54479) 2024-03-29T11:22:46.130 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:46.145+0000 7f3f6effd640 1 -- 172.21.15.84:0/4053315625 learned_addr learned my addr 172.21.15.84:0/4053315625 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-29T11:22:46.130 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:46.146+0000 7f3f577fe640 10 client.?.objecter ms_handle_connect 0x7f3f70074650 2024-03-29T11:22:46.130 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:46.146+0000 7f3f577fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:22:46.130 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:46.146+0000 7f3f577fe640 10 client.?.objecter ms_handle_connect 0x7f3f70141ff0 2024-03-29T11:22:46.130 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:46.146+0000 7f3f577fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:22:46.131 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:46.146+0000 7f3f577fe640 10 client.?.objecter ms_handle_connect 0x7f3f7014b870 2024-03-29T11:22:46.131 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:46.146+0000 7f3f577fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:22:46.131 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:46.146+0000 7f3f577fe640 1 -- 172.21.15.84:0/4053315625 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4124830384 0 0) 0x7f3f701d7980 con 0x7f3f70141ff0 2024-03-29T11:22:46.131 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:46.146+0000 7f3f577fe640 10 monclient(hunting): _negotiate 2024-03-29T11:22:46.131 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:46.146+0000 7f3f577fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:22:46.131 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:46.146+0000 7f3f577fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:22:46.131 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:46.146+0000 7f3f577fe640 10 monclient(hunting): my global_id is 4352 2024-03-29T11:22:46.131 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:46.146+0000 7f3f577fe640 1 -- 172.21.15.84:0/4053315625 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3f5c004eb0 con 0x7f3f70141ff0 2024-03-29T11:22:46.131 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:46.147+0000 7f3f577fe640 1 -- 172.21.15.84:0/4053315625 <== mon.0 v1:172.21.15.84:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2459290718 0 0) 0x7f3f70135d50 con 0x7f3f7014b870 2024-03-29T11:22:46.131 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:46.147+0000 7f3f577fe640 10 monclient(hunting): _negotiate 2024-03-29T11:22:46.131 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:46.147+0000 7f3f577fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:22:46.131 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:46.147+0000 7f3f577fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:22:46.131 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:46.147+0000 7f3f577fe640 10 monclient(hunting): my global_id is 4359 2024-03-29T11:22:46.131 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:46.147+0000 7f3f577fe640 1 -- 172.21.15.84:0/4053315625 --> v1:172.21.15.84:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3f701d7980 con 0x7f3f7014b870 2024-03-29T11:22:46.131 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:46.147+0000 7f3f577fe640 1 -- 172.21.15.84:0/4053315625 <== mon.2 v1:172.21.15.84:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 244188579 0 0) 0x7f3f5c004eb0 con 0x7f3f70141ff0 2024-03-29T11:22:46.132 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:46.147+0000 7f3f577fe640 1 -- 172.21.15.84:0/4053315625 --> v1:172.21.15.84:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3f70135d50 con 0x7f3f70141ff0 2024-03-29T11:22:46.132 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:46.147+0000 7f3f577fe640 1 -- 172.21.15.84:0/4053315625 <== mon.2 v1:172.21.15.84:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4236868694 0 0) 0x7f3f640017f0 con 0x7f3f70141ff0 2024-03-29T11:22:46.132 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:46.147+0000 7f3f577fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:46.132 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:46.147+0000 7f3f577fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:22:46.132 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:46.147+0000 7f3f577fe640 10 monclient(hunting): dump: 2024-03-29T11:22:46.132 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T11:22:46.132 INFO:teuthology.misc.health.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:22:46.132 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T11:18:53.864025+0000 2024-03-29T11:22:46.132 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T11:18:53.864025+0000 2024-03-29T11:22:46.132 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:46.132 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:22:46.132 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T11:22:46.132 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T11:22:46.132 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T11:22:46.132 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:22:46.132 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:46.147+0000 7f3f577fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:22:46.133 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:46.147+0000 7f3f577fe640 1 -- 172.21.15.84:0/4053315625 <== mon.0 v1:172.21.15.84:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1209753109 0 0) 0x7f3f701d7980 con 0x7f3f7014b870 2024-03-29T11:22:46.133 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:46.147+0000 7f3f577fe640 1 -- 172.21.15.84:0/4053315625 --> v1:172.21.15.84:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3f5c004eb0 con 0x7f3f7014b870 2024-03-29T11:22:46.133 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:46.147+0000 7f3f577fe640 1 -- 172.21.15.84:0/4053315625 <== mon.0 v1:172.21.15.84:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4236868694 0 0) 0x7f3f680018d0 con 0x7f3f7014b870 2024-03-29T11:22:46.133 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:46.147+0000 7f3f577fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:46.133 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:46.147+0000 7f3f577fe640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:22:46.133 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:46.147+0000 7f3f577fe640 10 monclient(hunting): dump: 2024-03-29T11:22:46.133 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T11:22:46.133 INFO:teuthology.misc.health.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:22:46.133 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T11:18:53.864025+0000 2024-03-29T11:22:46.133 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T11:18:53.864025+0000 2024-03-29T11:22:46.133 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:46.133 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:22:46.133 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T11:22:46.133 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T11:22:46.133 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T11:22:46.133 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:22:46.134 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:46.147+0000 7f3f577fe640 1 -- 172.21.15.84:0/4053315625 <== mon.2 v1:172.21.15.84:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4021862491 0 0) 0x7f3f70135d50 con 0x7f3f70141ff0 2024-03-29T11:22:46.134 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:46.147+0000 7f3f577fe640 1 -- 172.21.15.84:0/4053315625 >> v1:172.21.15.179:6789/0 conn(0x7f3f70074650 legacy=0x7f3f701d7270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:46.134 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:46.147+0000 7f3f577fe640 1 -- 172.21.15.84:0/4053315625 >> v1:172.21.15.84:6789/0 conn(0x7f3f7014b870 legacy=0x7f3f701e1a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:46.134 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:46.147+0000 7f3f577fe640 10 monclient: _finish_hunting 0 2024-03-29T11:22:46.134 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:46.147+0000 7f3f577fe640 1 monclient: found mon.c 2024-03-29T11:22:46.134 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:46.147+0000 7f3f577fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:22:46.134 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:46.147+0000 7f3f577fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-29T11:22:46.134 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:46.147+0000 7f3f577fe640 1 -- 172.21.15.84:0/4053315625 --> v1:172.21.15.84:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f701d7b10 con 0x7f3f70141ff0 2024-03-29T11:22:46.134 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:46.147+0000 7f3f577fe640 10 monclient: _finish_auth 0 2024-03-29T11:22:46.134 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:46.147+0000 7f3f577fe640 10 monclient: _check_auth_tickets 2024-03-29T11:22:46.134 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:46.147+0000 7f3f577fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:22:46.134 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:46.147+0000 7f3f7540f640 5 monclient: authenticate success, global_id 4352 2024-03-29T11:22:46.134 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:46.147+0000 7f3f7540f640 10 monclient: _renew_subs 2024-03-29T11:22:46.134 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:46.147+0000 7f3f7540f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-29T11:22:46.134 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:46.147+0000 7f3f7540f640 1 -- 172.21.15.84:0/4053315625 --> v1:172.21.15.84:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3f701d7d00 con 0x7f3f70141ff0 2024-03-29T11:22:46.135 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:46.148+0000 7f3f7540f640 10 client.4352.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:22:46.135 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:46.148+0000 7f3f577fe640 1 -- 172.21.15.84:0/4053315625 <== mon.2 v1:172.21.15.84:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3f64001950 con 0x7f3f70141ff0 2024-03-29T11:22:46.135 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:46.148+0000 7f3f577fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:22:46.135 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:46.148+0000 7f3f577fe640 1 -- 172.21.15.84:0/4053315625 <== mon.2 v1:172.21.15.84:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4236868694 0 0) 0x7f3f64003790 con 0x7f3f70141ff0 2024-03-29T11:22:46.135 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:46.148+0000 7f3f577fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:46.135 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:46.148+0000 7f3f577fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:22:46.135 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:46.148+0000 7f3f577fe640 10 monclient: dump: 2024-03-29T11:22:46.135 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T11:22:46.135 INFO:teuthology.misc.health.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:22:46.135 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T11:18:53.864025+0000 2024-03-29T11:22:46.135 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T11:18:53.864025+0000 2024-03-29T11:22:46.135 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:46.135 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:22:46.135 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T11:22:46.135 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T11:22:46.135 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T11:22:46.136 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:22:46.136 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:46.148+0000 7f3f7540f640 10 monclient: _renew_subs 2024-03-29T11:22:46.136 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:46.148+0000 7f3f7540f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-29T11:22:46.136 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:46.148+0000 7f3f7540f640 1 -- 172.21.15.84:0/4053315625 --> v1:172.21.15.84:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f3f701db160 con 0x7f3f70141ff0 2024-03-29T11:22:46.136 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:46.148+0000 7f3f7540f640 1 librados: init done 2024-03-29T11:22:46.136 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:46.149+0000 7f3f7540f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:22:46.136 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:46.149+0000 7f3f7540f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:22:46.136 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:46.149+0000 7f3f7540f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-29T11:22:46.136 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:46.149+0000 7f3f7540f640 1 -- 172.21.15.84:0/4053315625 --> v1:172.21.15.84:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3f70075f10 con 0x7f3f70141ff0 2024-03-29T11:22:46.136 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:46.150+0000 7f3f577fe640 1 -- 172.21.15.84:0/4053315625 <== mon.2 v1:172.21.15.84:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4033942909 0 0) 0x7f3f64019760 con 0x7f3f70141ff0 2024-03-29T11:22:46.136 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:46.150+0000 7f3f6d7fa640 4 set_mon_vals no callback set 2024-03-29T11:22:46.136 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:46.152+0000 7f3f577fe640 1 -- 172.21.15.84:0/4053315625 <== mon.2 v1:172.21.15.84:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2808139587 0 0) 0x7f3f64083750 con 0x7f3f70141ff0 2024-03-29T11:22:46.136 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:46.152+0000 7f3f577fe640 10 client.4352.objecter ms_dispatch 0x7f3f70000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:22:46.136 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:46.152+0000 7f3f6effd640 10 monclient: get_auth_request con 0x7f3f5c06ded0 auth_method 0 2024-03-29T11:22:46.137 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:46.152+0000 7f3f577fe640 3 client.4352.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:22:46.137 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:46.152+0000 7f3f577fe640 3 client.4352.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:22:46.137 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:46.152+0000 7f3f577fe640 20 client.4352.objecter dump_active .. 0 homeless 2024-03-29T11:22:46.137 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:46.153+0000 7f3f577fe640 10 client.4352.objecter ms_handle_connect 0x7f3f5c06ded0 2024-03-29T11:22:46.137 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:46.160+0000 7f3f577fe640 1 -- 172.21.15.84:0/4053315625 <== mon.2 v1:172.21.15.84:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f3f6404ee60 con 0x7f3f70141ff0 2024-03-29T11:22:46.137 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:46.160+0000 7f3f577fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:22:46.138 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:46.160+0000 7f3f577fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:22:46.454 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:46.477+0000 7f3f7540f640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:22:46.455 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:46.477+0000 7f3f7540f640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:22:46.455 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:46.477+0000 7f3f7540f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-29T11:22:46.455 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:46.477+0000 7f3f7540f640 1 -- 172.21.15.84:0/4053315625 --> v1:172.21.15.84:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3f7006e6a0 con 0x7f3f70141ff0 2024-03-29T11:22:46.458 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:46.481+0000 7f3f577fe640 1 -- 172.21.15.84:0/4053315625 <== mon.2 v1:172.21.15.84:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4209477549) 0x7f3f64050f60 con 0x7f3f70141ff0 2024-03-29T11:22:46.458 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:46.482+0000 7f3f577fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:22:46.459 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:46.482+0000 7f3f577fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:22:46.459 INFO:teuthology.misc.health.smithi084.stdout: 2024-03-29T11:22:46.459 INFO:teuthology.misc.health.smithi084.stdout:{"epoch":10,"fsid":"2c3e002f-9eb0-4860-853b-4ef1ef562213","created":"2024-03-29T11:19:32.029169+0000","modified":"2024-03-29T11:19:34.200299+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:34.200299+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"5c39e276-2034-4716-bbc0-57a634eefa92","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"cd580e99-109f-4bf1-88c0-a5987bc8b9ed","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"11e60a5a-051d-453f-a190-66387cb35592","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"2c329fbe-bbc4-4030-bf87-d711cad04679","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f4213d4a-bd36-4433-91ae-d7e2b02481d3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"f0cf44ea-6055-48af-805b-aed858102ff9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"d895eaa0-cc0b-4ee4-9b5a-2dfcd89a60b7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"9bc5a7bc-8f52-4db9-96d6-de5e68d52eaa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:22:46.462 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:46.485+0000 7f3f7540f640 10 librados: watch_flush enter 2024-03-29T11:22:46.463 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:46.486+0000 7f3f7540f640 10 librados: watch_flush exit 2024-03-29T11:22:46.463 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:46.486+0000 7f3f7540f640 20 client.4352.objecter shutdown clearing up homeless session... 2024-03-29T11:22:46.463 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:46.486+0000 7f3f7540f640 10 client.4352.objecter successfully canceled tick 2024-03-29T11:22:46.463 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:46.486+0000 7f3f7540f640 1 -- 172.21.15.84:0/4053315625 >> v1:172.21.15.179:6816/2981134980 conn(0x7f3f5c06ded0 legacy=0x7f3f5c070390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:46.463 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:46.486+0000 7f3f7540f640 10 monclient: shutdown 2024-03-29T11:22:46.463 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:46.486+0000 7f3f7540f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:22:46.463 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:46.486+0000 7f3f7540f640 1 -- 172.21.15.84:0/4053315625 >> v1:172.21.15.84:6790/0 conn(0x7f3f70141ff0 legacy=0x7f3f701da870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:46.464 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:46.486+0000 7f3f7540f640 1 -- 172.21.15.84:0/4053315625 shutdown_connections 2024-03-29T11:22:46.464 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:46.486+0000 7f3f7540f640 1 -- 172.21.15.84:0/4053315625 >> 172.21.15.84:0/4053315625 conn(0x7f3f700b3f40 msgr2=0x7f3f70138ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:22:46.464 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:46.487+0000 7f3f7540f640 1 -- 172.21.15.84:0/4053315625 shutdown_connections 2024-03-29T11:22:46.464 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:46.487+0000 7f3f7540f640 1 -- 172.21.15.84:0/4053315625 wait complete. 2024-03-29T11:22:46.464 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:46.487+0000 7f3f7540f640 1 librados: shutdown 2024-03-29T11:22:46.481 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:22:53.288 DEBUG:teuthology.orchestra.run.smithi084:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:22:53.440 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:53.462+0000 7f2202366640 10 monclient: get_monmap_and_config 2024-03-29T11:22:53.440 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:53.462+0000 7f2202366640 10 monclient: build_initial_monmap 2024-03-29T11:22:53.440 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:53.463+0000 7f2202366640 10 monclient: monmap: 2024-03-29T11:22:53.440 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-29T11:22:53.440 INFO:teuthology.misc.health.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:22:53.440 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T11:22:53.463913+0000 2024-03-29T11:22:53.440 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T11:22:53.463913+0000 2024-03-29T11:22:53.440 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-29T11:22:53.440 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:22:53.441 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-29T11:22:53.441 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.noname-c 2024-03-29T11:22:53.441 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-b 2024-03-29T11:22:53.441 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:22:53.442 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:53.463+0000 7f2202366640 1 Processor -- start 2024-03-29T11:22:53.442 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:53.464+0000 7f2202366640 1 -- start start 2024-03-29T11:22:53.442 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:53.464+0000 7f2202366640 10 monclient: init 2024-03-29T11:22:53.442 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:53.464+0000 7f2202366640 10 monclient: _reopen_session rank -1 2024-03-29T11:22:53.443 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:53.464+0000 7f2202366640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:22:53.443 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:53.464+0000 7f2202366640 10 monclient(hunting): picked mon.noname-a con 0x7f21fc0b4920 addr v1:172.21.15.84:6789/0 2024-03-29T11:22:53.443 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:53.464+0000 7f2202366640 10 monclient(hunting): picked mon.noname-b con 0x7f21fc0b4200 addr v1:172.21.15.84:6790/0 2024-03-29T11:22:53.443 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:53.465+0000 7f2202366640 10 monclient(hunting): picked mon.noname-c con 0x7f21fc13f9a0 addr v1:172.21.15.179:6789/0 2024-03-29T11:22:53.443 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:53.465+0000 7f2202366640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f21fc0b0930 con 0x7f21fc0b4920 2024-03-29T11:22:53.443 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:53.465+0000 7f2202366640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f21fc0af460 con 0x7f21fc13f9a0 2024-03-29T11:22:53.443 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:53.465+0000 7f2202366640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f21fc0aeda0 con 0x7f21fc0b4200 2024-03-29T11:22:53.443 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:53.465+0000 7f2202366640 10 monclient(hunting): _renew_subs 2024-03-29T11:22:53.443 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:53.465+0000 7f2202366640 10 monclient(hunting): authenticate will time out at 1034.759194s 2024-03-29T11:22:53.443 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:53.465+0000 7f2201b65640 1 --1- >> v1:172.21.15.179:6789/0 conn(0x7f21fc13f9a0 0x7f21fc14d8a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.179:6789/0 says I am v1:172.21.15.84:59190/0 (socket says 172.21.15.84:59190) 2024-03-29T11:22:53.443 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:53.465+0000 7f2201b65640 1 -- 172.21.15.84:0/1095576573 learned_addr learned my addr 172.21.15.84:0/1095576573 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-29T11:22:53.443 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:53.465+0000 7f21f3fff640 1 -- 172.21.15.84:0/1095576573 <== mon.1 v1:172.21.15.179:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 552253260 0 0) 0x7f21fc0af460 con 0x7f21fc13f9a0 2024-03-29T11:22:53.443 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:53.465+0000 7f21f3fff640 10 monclient(hunting): _negotiate 2024-03-29T11:22:53.443 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:53.465+0000 7f21f3fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:22:53.443 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:53.465+0000 7f21f3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:22:53.443 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:53.465+0000 7f21f3fff640 10 monclient(hunting): my global_id is 4324 2024-03-29T11:22:53.444 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:53.465+0000 7f21f3fff640 1 -- 172.21.15.84:0/1095576573 --> v1:172.21.15.179:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f21e4003520 con 0x7f21fc13f9a0 2024-03-29T11:22:53.444 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:53.466+0000 7f21f3fff640 1 -- 172.21.15.84:0/1095576573 <== mon.1 v1:172.21.15.179:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3826777492 0 0) 0x7f21e4003520 con 0x7f21fc13f9a0 2024-03-29T11:22:53.444 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:53.466+0000 7f21f3fff640 1 -- 172.21.15.84:0/1095576573 >> v1:172.21.15.84:6790/0 conn(0x7f21fc0b4200 legacy=0x7f21fc0b0fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:53.444 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:53.466+0000 7f21f3fff640 1 -- 172.21.15.84:0/1095576573 >> v1:172.21.15.84:6789/0 conn(0x7f21fc0b4920 legacy=0x7f21fc13f290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:53.444 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:53.466+0000 7f21f3fff640 10 monclient: _finish_hunting 0 2024-03-29T11:22:53.445 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:53.466+0000 7f21f3fff640 1 monclient: found mon.noname-c 2024-03-29T11:22:53.445 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:53.466+0000 7f21f3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:22:53.445 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:53.466+0000 7f21f3fff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.179:6789/0 2024-03-29T11:22:53.445 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:53.466+0000 7f21f3fff640 1 -- 172.21.15.84:0/1095576573 --> v1:172.21.15.179:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21fc0af980 con 0x7f21fc13f9a0 2024-03-29T11:22:53.445 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:53.466+0000 7f21f3fff640 10 monclient: _finish_auth 0 2024-03-29T11:22:53.445 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:53.466+0000 7f21f3fff640 10 monclient: _check_auth_tickets 2024-03-29T11:22:53.445 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:53.466+0000 7f21f3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:22:53.445 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:53.466+0000 7f21f3fff640 1 -- 172.21.15.84:0/1095576573 <== mon.1 v1:172.21.15.179:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4236868694 0 0) 0x7f21f4001690 con 0x7f21fc13f9a0 2024-03-29T11:22:53.445 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:53.466+0000 7f2202366640 5 monclient: authenticate success, global_id 4324 2024-03-29T11:22:53.445 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:53.466+0000 7f2202366640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:22:53.445 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:53.466+0000 7f21f3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:53.445 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:53.466+0000 7f21f3fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:22:53.445 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:53.466+0000 7f21f3fff640 10 monclient: dump: 2024-03-29T11:22:53.445 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T11:22:53.445 INFO:teuthology.misc.health.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:22:53.446 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T11:18:53.864025+0000 2024-03-29T11:22:53.446 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T11:18:53.864025+0000 2024-03-29T11:22:53.446 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:53.446 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:22:53.446 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T11:22:53.446 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T11:22:53.446 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T11:22:53.446 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:22:53.446 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:53.466+0000 7f2202366640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:22:53.448 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:53.467+0000 7f21f3fff640 1 -- 172.21.15.84:0/1095576573 <== mon.1 v1:172.21.15.179:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f21f40017f0 con 0x7f21fc13f9a0 2024-03-29T11:22:53.448 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:53.467+0000 7f21f3fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:22:53.448 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:53.467+0000 7f21f3fff640 1 -- 172.21.15.84:0/1095576573 <== mon.1 v1:172.21.15.179:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4236868694 0 0) 0x7f21f4001c00 con 0x7f21fc13f9a0 2024-03-29T11:22:53.448 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:53.467+0000 7f2202366640 10 monclient: get_monmap_and_config success 2024-03-29T11:22:53.448 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:53.467+0000 7f2202366640 4 set_mon_vals no callback set 2024-03-29T11:22:53.448 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:53.468+0000 7f21f3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:53.448 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:53.468+0000 7f21f3fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:22:53.448 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:53.468+0000 7f21f3fff640 10 monclient: dump: 2024-03-29T11:22:53.448 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T11:22:53.449 INFO:teuthology.misc.health.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:22:53.449 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T11:18:53.864025+0000 2024-03-29T11:22:53.449 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T11:18:53.864025+0000 2024-03-29T11:22:53.449 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:53.449 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:22:53.449 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T11:22:53.449 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T11:22:53.449 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T11:22:53.449 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:22:53.449 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:53.468+0000 7f2202366640 10 monclient: shutdown 2024-03-29T11:22:53.449 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:53.468+0000 7f2202366640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:22:53.449 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:53.468+0000 7f2202366640 1 -- 172.21.15.84:0/1095576573 >> v1:172.21.15.179:6789/0 conn(0x7f21fc13f9a0 legacy=0x7f21fc14d8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:53.449 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:53.468+0000 7f2202366640 1 -- 172.21.15.84:0/1095576573 shutdown_connections 2024-03-29T11:22:53.449 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:53.468+0000 7f2202366640 1 -- 172.21.15.84:0/1095576573 >> 172.21.15.84:0/1095576573 conn(0x7f21fc06d230 msgr2=0x7f21fc066a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:22:53.449 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:53.469+0000 7f2202366640 1 -- 172.21.15.84:0/1095576573 shutdown_connections 2024-03-29T11:22:53.449 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:53.469+0000 7f2202366640 1 -- 172.21.15.84:0/1095576573 wait complete. 2024-03-29T11:22:53.450 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:53.470+0000 7f2202366640 10 monclient: build_initial_monmap 2024-03-29T11:22:53.450 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:53.470+0000 7f2202366640 10 monclient: monmap: 2024-03-29T11:22:53.450 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-29T11:22:53.450 INFO:teuthology.misc.health.smithi084.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:22:53.450 INFO:teuthology.misc.health.smithi084.stderr:last_changed 0.000000 2024-03-29T11:22:53.450 INFO:teuthology.misc.health.smithi084.stderr:created 0.000000 2024-03-29T11:22:53.450 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-29T11:22:53.450 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:22:53.450 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-29T11:22:53.450 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.noname-b 2024-03-29T11:22:53.450 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-c 2024-03-29T11:22:53.450 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:22:53.450 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:53.470+0000 7f2202366640 1 librados: starting msgr at 2024-03-29T11:22:53.450 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:53.470+0000 7f2202366640 1 librados: starting objecter 2024-03-29T11:22:53.450 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:53.471+0000 7f2202366640 1 Processor -- start 2024-03-29T11:22:53.451 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:53.471+0000 7f2202366640 1 -- start start 2024-03-29T11:22:53.451 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:53.471+0000 7f2202366640 1 librados: setting wanted keys 2024-03-29T11:22:53.451 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:53.471+0000 7f2202366640 1 librados: calling monclient init 2024-03-29T11:22:53.451 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:53.471+0000 7f2202366640 10 monclient: init 2024-03-29T11:22:53.454 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:53.472+0000 7f2202366640 10 monclient: _reopen_session rank -1 2024-03-29T11:22:53.455 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:53.472+0000 7f2202366640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:22:53.455 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:53.472+0000 7f2202366640 10 monclient(hunting): picked mon.noname-b con 0x7f21fc0b4200 addr v1:172.21.15.179:6789/0 2024-03-29T11:22:53.455 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:53.472+0000 7f2202366640 10 monclient(hunting): picked mon.noname-c con 0x7f21fc0b4920 addr v1:172.21.15.84:6790/0 2024-03-29T11:22:53.455 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:53.472+0000 7f2201364640 1 --1- >> v1:172.21.15.179:6789/0 conn(0x7f21fc0b4200 0x7f21fc1dacb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.179:6789/0 says I am v1:172.21.15.84:59478/0 (socket says 172.21.15.84:59478) 2024-03-29T11:22:53.455 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:53.472+0000 7f2202366640 10 monclient(hunting): picked mon.noname-a con 0x7f21fc13f9a0 addr v1:172.21.15.84:6789/0 2024-03-29T11:22:53.455 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:53.472+0000 7f2202366640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f21fc1db3c0 con 0x7f21fc13f9a0 2024-03-29T11:22:53.455 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:53.472+0000 7f2202366640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f21fc0ab4f0 con 0x7f21fc0b4200 2024-03-29T11:22:53.455 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:53.472+0000 7f2202366640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f21fc1da4b0 con 0x7f21fc0b4920 2024-03-29T11:22:53.455 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:53.472+0000 7f2202366640 10 monclient(hunting): _renew_subs 2024-03-29T11:22:53.455 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:53.472+0000 7f2202366640 10 monclient(hunting): authenticate will time out at 1034.766935s 2024-03-29T11:22:53.455 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:53.472+0000 7f2201364640 1 -- 172.21.15.84:0/4122706253 learned_addr learned my addr 172.21.15.84:0/4122706253 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-29T11:22:53.455 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:53.473+0000 7f21f1ffb640 10 client.?.objecter ms_handle_connect 0x7f21fc0b4200 2024-03-29T11:22:53.455 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:53.473+0000 7f21f1ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:22:53.455 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:53.473+0000 7f21f1ffb640 10 client.?.objecter ms_handle_connect 0x7f21fc0b4920 2024-03-29T11:22:53.455 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:53.473+0000 7f21f1ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:22:53.456 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:53.473+0000 7f21f1ffb640 1 -- 172.21.15.84:0/4122706253 <== mon.1 v1:172.21.15.179:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2685583892 0 0) 0x7f21fc0ab4f0 con 0x7f21fc0b4200 2024-03-29T11:22:53.456 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:53.473+0000 7f21f1ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:22:53.456 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:53.473+0000 7f21f1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:22:53.456 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:53.473+0000 7f21f1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:22:53.456 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:53.473+0000 7f21f1ffb640 10 monclient(hunting): my global_id is 4327 2024-03-29T11:22:53.456 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:53.473+0000 7f21f1ffb640 1 -- 172.21.15.84:0/4122706253 --> v1:172.21.15.179:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f21e8004eb0 con 0x7f21fc0b4200 2024-03-29T11:22:53.456 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:53.473+0000 7f21f1ffb640 10 client.?.objecter ms_handle_connect 0x7f21fc13f9a0 2024-03-29T11:22:53.456 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:53.474+0000 7f21f1ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:22:53.456 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:53.474+0000 7f21f1ffb640 1 -- 172.21.15.84:0/4122706253 <== mon.1 v1:172.21.15.179:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 74723389 0 0) 0x7f21e8004eb0 con 0x7f21fc0b4200 2024-03-29T11:22:53.456 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:53.474+0000 7f21f1ffb640 1 -- 172.21.15.84:0/4122706253 --> v1:172.21.15.179:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f21fc0ab4f0 con 0x7f21fc0b4200 2024-03-29T11:22:53.456 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:53.474+0000 7f21f1ffb640 1 -- 172.21.15.84:0/4122706253 <== mon.1 v1:172.21.15.179:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4236868694 0 0) 0x7f21f8001a30 con 0x7f21fc0b4200 2024-03-29T11:22:53.456 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:53.474+0000 7f21f1ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:53.456 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:53.474+0000 7f21f1ffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:22:53.456 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:53.474+0000 7f21f1ffb640 10 monclient(hunting): dump: 2024-03-29T11:22:53.456 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T11:22:53.456 INFO:teuthology.misc.health.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:22:53.456 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T11:18:53.864025+0000 2024-03-29T11:22:53.457 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T11:18:53.864025+0000 2024-03-29T11:22:53.457 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:53.457 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:22:53.457 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T11:22:53.457 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T11:22:53.457 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T11:22:53.457 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:22:53.457 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:53.474+0000 7f21f1ffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:22:53.457 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:53.474+0000 7f21f1ffb640 1 -- 172.21.15.84:0/4122706253 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3621672665 0 0) 0x7f21fc1da4b0 con 0x7f21fc0b4920 2024-03-29T11:22:53.457 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:53.474+0000 7f21f1ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:22:53.457 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:53.474+0000 7f21f1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:22:53.457 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:53.474+0000 7f21f1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:22:53.457 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:53.474+0000 7f21f1ffb640 10 monclient(hunting): my global_id is 4358 2024-03-29T11:22:53.457 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:53.474+0000 7f21f1ffb640 1 -- 172.21.15.84:0/4122706253 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f21e8004eb0 con 0x7f21fc0b4920 2024-03-29T11:22:53.457 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:53.474+0000 7f21f1ffb640 1 -- 172.21.15.84:0/4122706253 <== mon.1 v1:172.21.15.179:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2558876516 0 0) 0x7f21fc0ab4f0 con 0x7f21fc0b4200 2024-03-29T11:22:53.457 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:53.474+0000 7f21f1ffb640 1 -- 172.21.15.84:0/4122706253 >> v1:172.21.15.84:6790/0 conn(0x7f21fc0b4920 legacy=0x7f21fc1391a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:53.458 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:53.474+0000 7f21f1ffb640 1 -- 172.21.15.84:0/4122706253 >> v1:172.21.15.84:6789/0 conn(0x7f21fc13f9a0 legacy=0x7f21fc1e2850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:53.458 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:53.474+0000 7f21f1ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:22:53.458 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:53.474+0000 7f21f1ffb640 1 monclient: found mon.b 2024-03-29T11:22:53.458 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:53.474+0000 7f21f1ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:22:53.458 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:53.474+0000 7f21f1ffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-29T11:22:53.458 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:53.474+0000 7f21f1ffb640 1 -- 172.21.15.84:0/4122706253 --> v1:172.21.15.179:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21fc1da790 con 0x7f21fc0b4200 2024-03-29T11:22:53.458 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:53.474+0000 7f21f1ffb640 10 monclient: _finish_auth 0 2024-03-29T11:22:53.458 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:53.475+0000 7f21f1ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:22:53.458 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:53.475+0000 7f21f1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:22:53.458 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:53.475+0000 7f2202366640 5 monclient: authenticate success, global_id 4327 2024-03-29T11:22:53.458 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:53.475+0000 7f2202366640 10 monclient: _renew_subs 2024-03-29T11:22:53.458 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:53.475+0000 7f2202366640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-29T11:22:53.458 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:53.475+0000 7f2202366640 1 -- 172.21.15.84:0/4122706253 --> v1:172.21.15.179:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f21fc1398b0 con 0x7f21fc0b4200 2024-03-29T11:22:53.458 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:53.475+0000 7f21f1ffb640 1 -- 172.21.15.84:0/4122706253 <== mon.1 v1:172.21.15.179:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f21f8001cb0 con 0x7f21fc0b4200 2024-03-29T11:22:53.458 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:53.475+0000 7f21f1ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:22:53.459 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:53.475+0000 7f21f37fe640 4 set_mon_vals no callback set 2024-03-29T11:22:53.459 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:53.475+0000 7f2202366640 10 client.4327.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:22:53.459 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:53.475+0000 7f21f1ffb640 1 -- 172.21.15.84:0/4122706253 <== mon.1 v1:172.21.15.179:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4236868694 0 0) 0x7f21f8003a60 con 0x7f21fc0b4200 2024-03-29T11:22:53.459 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:53.475+0000 7f21f1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:53.459 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:53.475+0000 7f21f1ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:22:53.459 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:53.475+0000 7f21f1ffb640 10 monclient: dump: 2024-03-29T11:22:53.459 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T11:22:53.459 INFO:teuthology.misc.health.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:22:53.459 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T11:18:53.864025+0000 2024-03-29T11:22:53.459 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T11:18:53.864025+0000 2024-03-29T11:22:53.459 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:53.459 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:22:53.459 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T11:22:53.459 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T11:22:53.459 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T11:22:53.459 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:22:53.460 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:53.475+0000 7f2202366640 10 monclient: _renew_subs 2024-03-29T11:22:53.460 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:53.475+0000 7f2202366640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-29T11:22:53.460 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:53.475+0000 7f2202366640 1 -- 172.21.15.84:0/4122706253 --> v1:172.21.15.179:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f21fc0a9930 con 0x7f21fc0b4200 2024-03-29T11:22:53.460 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:53.475+0000 7f2202366640 1 librados: init done 2024-03-29T11:22:53.460 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:53.476+0000 7f2202366640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:22:53.461 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:53.477+0000 7f2202366640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:22:53.461 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:53.477+0000 7f2202366640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-29T11:22:53.461 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:53.477+0000 7f2202366640 1 -- 172.21.15.84:0/4122706253 --> v1:172.21.15.179:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f21fc04f730 con 0x7f21fc0b4200 2024-03-29T11:22:53.461 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:53.478+0000 7f21f1ffb640 1 -- 172.21.15.84:0/4122706253 <== mon.1 v1:172.21.15.179:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4033942909 0 0) 0x7f21f8003bc0 con 0x7f21fc0b4200 2024-03-29T11:22:53.461 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:53.479+0000 7f2200b63640 10 monclient: get_auth_request con 0x7f21e806dd70 auth_method 0 2024-03-29T11:22:53.461 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:53.479+0000 7f21f1ffb640 1 -- 172.21.15.84:0/4122706253 <== mon.1 v1:172.21.15.179:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2808139587 0 0) 0x7f21f8084b20 con 0x7f21fc0b4200 2024-03-29T11:22:53.461 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:53.479+0000 7f21f1ffb640 10 client.4327.objecter ms_dispatch 0x7f21fc000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:22:53.461 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:53.479+0000 7f21f1ffb640 3 client.4327.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:22:53.461 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:53.479+0000 7f21f1ffb640 3 client.4327.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:22:53.461 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:53.479+0000 7f21f1ffb640 20 client.4327.objecter dump_active .. 0 homeless 2024-03-29T11:22:53.461 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:53.479+0000 7f21f1ffb640 10 client.4327.objecter ms_handle_connect 0x7f21e806dd70 2024-03-29T11:22:53.463 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:53.485+0000 7f21f1ffb640 1 -- 172.21.15.84:0/4122706253 <== mon.1 v1:172.21.15.179:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f21f8050400 con 0x7f21fc0b4200 2024-03-29T11:22:53.463 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:53.485+0000 7f21f1ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:22:53.463 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:53.486+0000 7f21f1ffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:22:53.905 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:53.928+0000 7f2202366640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:22:53.905 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:53.928+0000 7f2202366640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:22:53.905 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:53.928+0000 7f2202366640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-29T11:22:53.905 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:53.928+0000 7f2202366640 1 -- 172.21.15.84:0/4122706253 --> v1:172.21.15.179:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21fc1da920 con 0x7f21fc0b4200 2024-03-29T11:22:53.906 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:53.929+0000 7f21f1ffb640 1 -- 172.21.15.84:0/4122706253 <== mon.1 v1:172.21.15.179:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4209477549) 0x7f21f8052330 con 0x7f21fc0b4200 2024-03-29T11:22:53.906 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:53.929+0000 7f21f1ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:22:53.906 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:53.929+0000 7f21f1ffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:22:53.906 INFO:teuthology.misc.health.smithi084.stdout: 2024-03-29T11:22:53.906 INFO:teuthology.misc.health.smithi084.stdout:{"epoch":10,"fsid":"2c3e002f-9eb0-4860-853b-4ef1ef562213","created":"2024-03-29T11:19:32.029169+0000","modified":"2024-03-29T11:19:34.200299+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:34.200299+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"5c39e276-2034-4716-bbc0-57a634eefa92","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"cd580e99-109f-4bf1-88c0-a5987bc8b9ed","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"11e60a5a-051d-453f-a190-66387cb35592","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"2c329fbe-bbc4-4030-bf87-d711cad04679","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f4213d4a-bd36-4433-91ae-d7e2b02481d3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"f0cf44ea-6055-48af-805b-aed858102ff9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"d895eaa0-cc0b-4ee4-9b5a-2dfcd89a60b7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"9bc5a7bc-8f52-4db9-96d6-de5e68d52eaa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:22:53.909 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:53.932+0000 7f2202366640 10 librados: watch_flush enter 2024-03-29T11:22:53.910 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:53.932+0000 7f2202366640 10 librados: watch_flush exit 2024-03-29T11:22:53.910 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:53.932+0000 7f2202366640 20 client.4327.objecter shutdown clearing up homeless session... 2024-03-29T11:22:53.910 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:53.932+0000 7f2202366640 10 client.4327.objecter successfully canceled tick 2024-03-29T11:22:53.910 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:53.933+0000 7f2202366640 1 -- 172.21.15.84:0/4122706253 >> v1:172.21.15.179:6816/2981134980 conn(0x7f21e806dd70 legacy=0x7f21e8070230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:53.910 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:53.933+0000 7f2202366640 10 monclient: shutdown 2024-03-29T11:22:53.911 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:53.933+0000 7f2202366640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:22:53.911 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:53.933+0000 7f2202366640 1 -- 172.21.15.84:0/4122706253 >> v1:172.21.15.179:6789/0 conn(0x7f21fc0b4200 legacy=0x7f21fc1dacb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:53.911 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:53.933+0000 7f2202366640 1 -- 172.21.15.84:0/4122706253 shutdown_connections 2024-03-29T11:22:53.911 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:53.933+0000 7f2202366640 1 -- 172.21.15.84:0/4122706253 >> 172.21.15.84:0/4122706253 conn(0x7f21fc06d230 msgr2=0x7f21fc069b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:22:53.911 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:53.933+0000 7f2202366640 1 -- 172.21.15.84:0/4122706253 shutdown_connections 2024-03-29T11:22:53.911 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:53.933+0000 7f2202366640 1 -- 172.21.15.84:0/4122706253 wait complete. 2024-03-29T11:22:53.911 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:22:53.933+0000 7f2202366640 1 librados: shutdown 2024-03-29T11:22:53.926 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:23:00.734 DEBUG:teuthology.orchestra.run.smithi084:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:23:00.885 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:00.907+0000 7ff2fe739640 10 monclient: get_monmap_and_config 2024-03-29T11:23:00.885 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:00.907+0000 7ff2fe739640 10 monclient: build_initial_monmap 2024-03-29T11:23:00.885 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:00.907+0000 7ff2fe739640 10 monclient: monmap: 2024-03-29T11:23:00.885 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-29T11:23:00.885 INFO:teuthology.misc.health.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:23:00.885 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T11:23:00.908412+0000 2024-03-29T11:23:00.885 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T11:23:00.908412+0000 2024-03-29T11:23:00.885 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-29T11:23:00.885 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:23:00.885 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-29T11:23:00.886 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.noname-c 2024-03-29T11:23:00.886 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-b 2024-03-29T11:23:00.886 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:23:00.887 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:00.908+0000 7ff2fe739640 1 Processor -- start 2024-03-29T11:23:00.887 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:00.908+0000 7ff2fe739640 1 -- start start 2024-03-29T11:23:00.887 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:00.908+0000 7ff2fe739640 10 monclient: init 2024-03-29T11:23:00.887 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:00.909+0000 7ff2fe739640 10 monclient: _reopen_session rank -1 2024-03-29T11:23:00.887 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:00.909+0000 7ff2fe739640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:23:00.887 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:00.909+0000 7ff2fe739640 10 monclient(hunting): picked mon.noname-b con 0x7ff2f81422b0 addr v1:172.21.15.84:6790/0 2024-03-29T11:23:00.887 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:00.909+0000 7ff2fe739640 10 monclient(hunting): picked mon.noname-a con 0x7ff2f8142690 addr v1:172.21.15.84:6789/0 2024-03-29T11:23:00.887 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:00.909+0000 7ff2fe739640 10 monclient(hunting): picked mon.noname-c con 0x7ff2f80b17d0 addr v1:172.21.15.179:6789/0 2024-03-29T11:23:00.887 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:00.909+0000 7ff2fe739640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff2f8142c80 con 0x7ff2f8142690 2024-03-29T11:23:00.887 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:00.909+0000 7ff2fe739640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff2f80aae40 con 0x7ff2f80b17d0 2024-03-29T11:23:00.887 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:00.909+0000 7ff2fe739640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff2f813fbd0 con 0x7ff2f81422b0 2024-03-29T11:23:00.888 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:00.909+0000 7ff2fe739640 10 monclient(hunting): _renew_subs 2024-03-29T11:23:00.888 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:00.909+0000 7ff2fe739640 10 monclient(hunting): authenticate will time out at 1042.203753s 2024-03-29T11:23:00.888 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:00.909+0000 7ff2f77fe640 1 --1- >> v1:172.21.15.84:6789/0 conn(0x7ff2f8142690 0x7ff2f80b2590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.84:6789/0 says I am v1:172.21.15.84:55848/0 (socket says 172.21.15.84:55848) 2024-03-29T11:23:00.888 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:00.909+0000 7ff2f77fe640 1 -- 172.21.15.84:0/3510340252 learned_addr learned my addr 172.21.15.84:0/3510340252 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-29T11:23:00.889 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:00.910+0000 7ff2f6ffd640 1 -- 172.21.15.84:0/3510340252 <== mon.0 v1:172.21.15.84:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1989988345 0 0) 0x7ff2f8142c80 con 0x7ff2f8142690 2024-03-29T11:23:00.889 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:00.910+0000 7ff2f6ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:23:00.890 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:00.910+0000 7ff2f6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:23:00.890 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:00.910+0000 7ff2f6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:23:00.890 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:00.910+0000 7ff2f6ffd640 10 monclient(hunting): my global_id is 4368 2024-03-29T11:23:00.890 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:00.910+0000 7ff2f6ffd640 1 -- 172.21.15.84:0/3510340252 --> v1:172.21.15.84:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff2e4003520 con 0x7ff2f8142690 2024-03-29T11:23:00.890 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:00.910+0000 7ff2f6ffd640 1 -- 172.21.15.84:0/3510340252 <== mon.1 v1:172.21.15.179:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 779634778 0 0) 0x7ff2f80aae40 con 0x7ff2f80b17d0 2024-03-29T11:23:00.890 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:00.910+0000 7ff2f6ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:23:00.890 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:00.910+0000 7ff2f6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:23:00.890 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:00.910+0000 7ff2f6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:23:00.890 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:00.910+0000 7ff2f6ffd640 10 monclient(hunting): my global_id is 4330 2024-03-29T11:23:00.890 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:00.910+0000 7ff2f6ffd640 1 -- 172.21.15.84:0/3510340252 --> v1:172.21.15.179:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff2f8142c80 con 0x7ff2f80b17d0 2024-03-29T11:23:00.890 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:00.910+0000 7ff2f6ffd640 1 -- 172.21.15.84:0/3510340252 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 659436396 0 0) 0x7ff2f813fbd0 con 0x7ff2f81422b0 2024-03-29T11:23:00.890 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:00.910+0000 7ff2f6ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:23:00.890 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:00.910+0000 7ff2f6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:23:00.890 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:00.910+0000 7ff2f6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:23:00.891 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:00.910+0000 7ff2f6ffd640 10 monclient(hunting): my global_id is 4361 2024-03-29T11:23:00.891 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:00.910+0000 7ff2f6ffd640 1 -- 172.21.15.84:0/3510340252 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff2f80aae40 con 0x7ff2f81422b0 2024-03-29T11:23:00.891 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:00.910+0000 7ff2f6ffd640 1 -- 172.21.15.84:0/3510340252 <== mon.0 v1:172.21.15.84:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3495485124 0 0) 0x7ff2e4003520 con 0x7ff2f8142690 2024-03-29T11:23:00.891 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:00.910+0000 7ff2f6ffd640 1 -- 172.21.15.84:0/3510340252 >> v1:172.21.15.84:6790/0 conn(0x7ff2f81422b0 legacy=0x7ff2f813f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:00.891 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:00.910+0000 7ff2f6ffd640 1 -- 172.21.15.84:0/3510340252 >> v1:172.21.15.179:6789/0 conn(0x7ff2f80b17d0 legacy=0x7ff2f80b2ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:00.891 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:00.911+0000 7ff2f6ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:23:00.891 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:00.911+0000 7ff2f6ffd640 1 monclient: found mon.noname-a 2024-03-29T11:23:00.891 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:00.911+0000 7ff2f6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:23:00.891 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:00.911+0000 7ff2f6ffd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.84:6789/0 2024-03-29T11:23:00.891 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:00.911+0000 7ff2f6ffd640 1 -- 172.21.15.84:0/3510340252 --> v1:172.21.15.84:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2f8151940 con 0x7ff2f8142690 2024-03-29T11:23:00.891 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:00.911+0000 7ff2f6ffd640 10 monclient: _finish_auth 0 2024-03-29T11:23:00.892 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:00.911+0000 7ff2f6ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:23:00.892 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:00.911+0000 7ff2f6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:23:00.892 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:00.911+0000 7ff2f6ffd640 1 -- 172.21.15.84:0/3510340252 <== mon.0 v1:172.21.15.84:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4236868694 0 0) 0x7ff2e8001690 con 0x7ff2f8142690 2024-03-29T11:23:00.892 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:00.911+0000 7ff2f6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:23:00.892 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:00.911+0000 7ff2f6ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:23:00.892 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:00.911+0000 7ff2f6ffd640 10 monclient: dump: 2024-03-29T11:23:00.892 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T11:23:00.892 INFO:teuthology.misc.health.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:23:00.892 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T11:18:53.864025+0000 2024-03-29T11:23:00.892 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T11:18:53.864025+0000 2024-03-29T11:23:00.892 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T11:23:00.892 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:23:00.892 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T11:23:00.892 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T11:23:00.892 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T11:23:00.893 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:23:00.893 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:00.911+0000 7ff2fe739640 5 monclient: authenticate success, global_id 4368 2024-03-29T11:23:00.893 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:00.911+0000 7ff2fe739640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:23:00.893 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:00.911+0000 7ff2f6ffd640 1 -- 172.21.15.84:0/3510340252 <== mon.0 v1:172.21.15.84:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff2e80017f0 con 0x7ff2f8142690 2024-03-29T11:23:00.893 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:00.911+0000 7ff2f6ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:23:00.893 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:00.911+0000 7ff2fe739640 10 monclient: get_monmap_and_config success 2024-03-29T11:23:00.893 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:00.911+0000 7ff2fe739640 4 set_mon_vals no callback set 2024-03-29T11:23:00.897 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:00.912+0000 7ff2fe739640 10 monclient: shutdown 2024-03-29T11:23:00.897 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:00.912+0000 7ff2fe739640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:23:00.897 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:00.912+0000 7ff2fe739640 1 -- 172.21.15.84:0/3510340252 >> v1:172.21.15.84:6789/0 conn(0x7ff2f8142690 legacy=0x7ff2f80b2590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:00.898 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:00.913+0000 7ff2fe739640 1 -- 172.21.15.84:0/3510340252 shutdown_connections 2024-03-29T11:23:00.898 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:00.913+0000 7ff2fe739640 1 -- 172.21.15.84:0/3510340252 >> 172.21.15.84:0/3510340252 conn(0x7ff2f8066a00 msgr2=0x7ff2f8066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:23:00.898 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:00.913+0000 7ff2fe739640 1 -- 172.21.15.84:0/3510340252 shutdown_connections 2024-03-29T11:23:00.898 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:00.913+0000 7ff2fe739640 1 -- 172.21.15.84:0/3510340252 wait complete. 2024-03-29T11:23:00.898 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:00.914+0000 7ff2fe739640 10 monclient: build_initial_monmap 2024-03-29T11:23:00.898 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:00.914+0000 7ff2fe739640 10 monclient: monmap: 2024-03-29T11:23:00.898 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-29T11:23:00.898 INFO:teuthology.misc.health.smithi084.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:23:00.898 INFO:teuthology.misc.health.smithi084.stderr:last_changed 0.000000 2024-03-29T11:23:00.898 INFO:teuthology.misc.health.smithi084.stderr:created 0.000000 2024-03-29T11:23:00.899 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-29T11:23:00.899 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:23:00.899 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-29T11:23:00.899 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.noname-b 2024-03-29T11:23:00.899 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-c 2024-03-29T11:23:00.899 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:23:00.899 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:00.914+0000 7ff2fe739640 1 librados: starting msgr at 2024-03-29T11:23:00.899 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:00.914+0000 7ff2fe739640 1 librados: starting objecter 2024-03-29T11:23:00.899 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:00.914+0000 7ff2fe739640 1 Processor -- start 2024-03-29T11:23:00.899 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:00.914+0000 7ff2fe739640 1 -- start start 2024-03-29T11:23:00.899 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:00.914+0000 7ff2fe739640 1 librados: setting wanted keys 2024-03-29T11:23:00.899 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:00.914+0000 7ff2fe739640 1 librados: calling monclient init 2024-03-29T11:23:00.899 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:00.914+0000 7ff2fe739640 10 monclient: init 2024-03-29T11:23:00.899 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:00.915+0000 7ff2fe739640 10 monclient: _reopen_session rank -1 2024-03-29T11:23:00.899 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:00.915+0000 7ff2fe739640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:23:00.900 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:00.915+0000 7ff2fe739640 10 monclient(hunting): picked mon.noname-a con 0x7ff2f80b17d0 addr v1:172.21.15.84:6789/0 2024-03-29T11:23:00.900 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:00.915+0000 7ff2fe739640 10 monclient(hunting): picked mon.noname-c con 0x7ff2f81422b0 addr v1:172.21.15.84:6790/0 2024-03-29T11:23:00.900 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:00.915+0000 7ff2fe739640 10 monclient(hunting): picked mon.noname-b con 0x7ff2f8142690 addr v1:172.21.15.179:6789/0 2024-03-29T11:23:00.900 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:00.915+0000 7ff2fe739640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff2f80b7110 con 0x7ff2f80b17d0 2024-03-29T11:23:00.900 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:00.915+0000 7ff2fe739640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff2f812dc40 con 0x7ff2f8142690 2024-03-29T11:23:00.900 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:00.915+0000 7ff2fe739640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff2f80ad070 con 0x7ff2f81422b0 2024-03-29T11:23:00.900 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:00.915+0000 7ff2fe739640 10 monclient(hunting): _renew_subs 2024-03-29T11:23:00.900 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:00.915+0000 7ff2fe739640 10 monclient(hunting): authenticate will time out at 1042.209816s 2024-03-29T11:23:00.900 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:00.915+0000 7ff2f7fff640 1 --1- >> v1:172.21.15.84:6789/0 conn(0x7ff2f80b17d0 0x7ff2f81da890 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.84:6789/0 says I am v1:172.21.15.84:59258/0 (socket says 172.21.15.84:59258) 2024-03-29T11:23:00.900 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:00.915+0000 7ff2f7fff640 1 -- 172.21.15.84:0/2371321979 learned_addr learned my addr 172.21.15.84:0/2371321979 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-29T11:23:00.900 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:00.916+0000 7ff2f4ff9640 10 client.?.objecter ms_handle_connect 0x7ff2f8142690 2024-03-29T11:23:00.900 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:00.916+0000 7ff2f4ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:23:00.900 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:00.916+0000 7ff2f4ff9640 10 client.?.objecter ms_handle_connect 0x7ff2f80b17d0 2024-03-29T11:23:00.900 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:00.916+0000 7ff2f4ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:23:00.900 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:00.916+0000 7ff2f4ff9640 10 client.?.objecter ms_handle_connect 0x7ff2f81422b0 2024-03-29T11:23:00.901 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:00.916+0000 7ff2f4ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:23:00.901 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:00.916+0000 7ff2f4ff9640 1 -- 172.21.15.84:0/2371321979 <== mon.1 v1:172.21.15.179:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 628425949 0 0) 0x7ff2f812dc40 con 0x7ff2f8142690 2024-03-29T11:23:00.901 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:00.916+0000 7ff2f4ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:23:00.901 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:00.916+0000 7ff2f4ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:23:00.901 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:00.916+0000 7ff2f4ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:23:00.901 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:00.916+0000 7ff2f4ff9640 10 monclient(hunting): my global_id is 4333 2024-03-29T11:23:00.901 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:00.916+0000 7ff2f4ff9640 1 -- 172.21.15.84:0/2371321979 --> v1:172.21.15.179:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff2d8004eb0 con 0x7ff2f8142690 2024-03-29T11:23:00.901 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:00.916+0000 7ff2f4ff9640 1 -- 172.21.15.84:0/2371321979 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 940260669 0 0) 0x7ff2f80ad070 con 0x7ff2f81422b0 2024-03-29T11:23:00.901 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:00.916+0000 7ff2f4ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:23:00.901 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:00.916+0000 7ff2f4ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:23:00.901 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:00.916+0000 7ff2f4ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:23:00.901 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:00.916+0000 7ff2f4ff9640 10 monclient(hunting): my global_id is 4364 2024-03-29T11:23:00.901 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:00.916+0000 7ff2f4ff9640 1 -- 172.21.15.84:0/2371321979 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff2f812dc40 con 0x7ff2f81422b0 2024-03-29T11:23:00.901 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:00.916+0000 7ff2f4ff9640 1 -- 172.21.15.84:0/2371321979 <== mon.1 v1:172.21.15.179:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2794993385 0 0) 0x7ff2d8004eb0 con 0x7ff2f8142690 2024-03-29T11:23:00.901 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:00.917+0000 7ff2f4ff9640 1 -- 172.21.15.84:0/2371321979 --> v1:172.21.15.179:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff2f80ad070 con 0x7ff2f8142690 2024-03-29T11:23:00.902 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:00.917+0000 7ff2f4ff9640 1 -- 172.21.15.84:0/2371321979 <== mon.1 v1:172.21.15.179:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4236868694 0 0) 0x7ff2f0001a30 con 0x7ff2f8142690 2024-03-29T11:23:00.902 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:00.917+0000 7ff2f4ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:23:00.902 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:00.917+0000 7ff2f4ff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:23:00.902 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:00.917+0000 7ff2f4ff9640 10 monclient(hunting): dump: 2024-03-29T11:23:00.902 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T11:23:00.902 INFO:teuthology.misc.health.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:23:00.902 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T11:18:53.864025+0000 2024-03-29T11:23:00.902 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T11:18:53.864025+0000 2024-03-29T11:23:00.902 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T11:23:00.902 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:23:00.902 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T11:23:00.902 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T11:23:00.903 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T11:23:00.903 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:23:00.903 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:00.917+0000 7ff2f4ff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:23:00.903 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:00.917+0000 7ff2f4ff9640 1 -- 172.21.15.84:0/2371321979 <== mon.1 v1:172.21.15.179:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2998689509 0 0) 0x7ff2f80ad070 con 0x7ff2f8142690 2024-03-29T11:23:00.903 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:00.917+0000 7ff2f4ff9640 1 -- 172.21.15.84:0/2371321979 >> v1:172.21.15.84:6790/0 conn(0x7ff2f81422b0 legacy=0x7ff2f812ce20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:00.903 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:00.917+0000 7ff2f4ff9640 1 -- 172.21.15.84:0/2371321979 >> v1:172.21.15.84:6789/0 conn(0x7ff2f80b17d0 legacy=0x7ff2f81da890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:00.903 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:00.917+0000 7ff2f4ff9640 10 monclient: _finish_hunting 0 2024-03-29T11:23:00.903 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:00.917+0000 7ff2f4ff9640 1 monclient: found mon.b 2024-03-29T11:23:00.903 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:00.917+0000 7ff2f4ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:23:00.903 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:00.917+0000 7ff2f4ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-29T11:23:00.903 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:00.917+0000 7ff2f4ff9640 1 -- 172.21.15.84:0/2371321979 --> v1:172.21.15.179:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2f81dafa0 con 0x7ff2f8142690 2024-03-29T11:23:00.903 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:00.917+0000 7ff2f4ff9640 10 monclient: _finish_auth 0 2024-03-29T11:23:00.903 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:00.917+0000 7ff2f4ff9640 10 monclient: _check_auth_tickets 2024-03-29T11:23:00.903 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:00.917+0000 7ff2f4ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:23:00.903 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:00.917+0000 7ff2fe739640 5 monclient: authenticate success, global_id 4333 2024-03-29T11:23:00.904 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:00.917+0000 7ff2fe739640 10 monclient: _renew_subs 2024-03-29T11:23:00.904 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:00.917+0000 7ff2fe739640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-29T11:23:00.904 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:00.917+0000 7ff2fe739640 1 -- 172.21.15.84:0/2371321979 --> v1:172.21.15.179:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff2f812def0 con 0x7ff2f8142690 2024-03-29T11:23:00.904 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:00.917+0000 7ff2fe739640 10 client.4333.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:23:00.904 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:00.917+0000 7ff2fe739640 10 monclient: _renew_subs 2024-03-29T11:23:00.904 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:00.917+0000 7ff2fe739640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-29T11:23:00.904 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:00.917+0000 7ff2fe739640 1 -- 172.21.15.84:0/2371321979 --> v1:172.21.15.179:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff2f8150a30 con 0x7ff2f8142690 2024-03-29T11:23:00.904 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:00.917+0000 7ff2fe739640 1 librados: init done 2024-03-29T11:23:00.904 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:00.917+0000 7ff2f4ff9640 1 -- 172.21.15.84:0/2371321979 <== mon.1 v1:172.21.15.179:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff2f0003970 con 0x7ff2f8142690 2024-03-29T11:23:00.904 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:00.917+0000 7ff2f4ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:23:00.904 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:00.917+0000 7ff2f4ff9640 1 -- 172.21.15.84:0/2371321979 <== mon.1 v1:172.21.15.179:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4236868694 0 0) 0x7ff2f0003ae0 con 0x7ff2f8142690 2024-03-29T11:23:00.904 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:00.917+0000 7ff2f4ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:23:00.904 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:00.917+0000 7ff2f4ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:23:00.904 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:00.917+0000 7ff2f4ff9640 10 monclient: dump: 2024-03-29T11:23:00.904 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T11:23:00.905 INFO:teuthology.misc.health.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:23:00.905 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T11:18:53.864025+0000 2024-03-29T11:23:00.905 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T11:18:53.864025+0000 2024-03-29T11:23:00.905 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T11:23:00.905 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:23:00.905 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T11:23:00.905 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T11:23:00.905 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T11:23:00.905 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:23:00.905 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:00.917+0000 7ff2f67fc640 4 set_mon_vals no callback set 2024-03-29T11:23:00.905 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:00.919+0000 7ff2fe739640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:23:00.905 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:00.919+0000 7ff2fe739640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:23:00.906 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:00.919+0000 7ff2fe739640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-29T11:23:00.906 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:00.919+0000 7ff2fe739640 1 -- 172.21.15.84:0/2371321979 --> v1:172.21.15.179:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff2dc005080 con 0x7ff2f8142690 2024-03-29T11:23:00.907 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:00.920+0000 7ff2f4ff9640 1 -- 172.21.15.84:0/2371321979 <== mon.1 v1:172.21.15.179:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4033942909 0 0) 0x7ff2f0019ab0 con 0x7ff2f8142690 2024-03-29T11:23:00.907 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:00.921+0000 7ff2f4ff9640 1 -- 172.21.15.84:0/2371321979 <== mon.1 v1:172.21.15.179:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2808139587 0 0) 0x7ff2f0083990 con 0x7ff2f8142690 2024-03-29T11:23:00.907 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:00.921+0000 7ff2f4ff9640 10 client.4333.objecter ms_dispatch 0x7ff2f8000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:23:00.907 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:00.921+0000 7ff2f4ff9640 3 client.4333.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:23:00.907 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:00.921+0000 7ff2f4ff9640 3 client.4333.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:23:00.907 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:00.921+0000 7ff2f4ff9640 20 client.4333.objecter dump_active .. 0 homeless 2024-03-29T11:23:00.907 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:00.921+0000 7ff2f7fff640 10 monclient: get_auth_request con 0x7ff2d806dd60 auth_method 0 2024-03-29T11:23:00.907 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:00.921+0000 7ff2f4ff9640 10 client.4333.objecter ms_handle_connect 0x7ff2d806dd60 2024-03-29T11:23:00.908 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:00.931+0000 7ff2f4ff9640 1 -- 172.21.15.84:0/2371321979 <== mon.1 v1:172.21.15.179:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7ff2f004f0a0 con 0x7ff2f8142690 2024-03-29T11:23:00.908 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:00.931+0000 7ff2f4ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:23:00.909 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:00.931+0000 7ff2f4ff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:23:01.260 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:01.282+0000 7ff2fe739640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:23:01.260 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:01.282+0000 7ff2fe739640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:23:01.260 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:01.282+0000 7ff2fe739640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-29T11:23:01.260 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:01.282+0000 7ff2fe739640 1 -- 172.21.15.84:0/2371321979 --> v1:172.21.15.179:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff2dc005330 con 0x7ff2f8142690 2024-03-29T11:23:01.261 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:01.283+0000 7ff2f4ff9640 1 -- 172.21.15.84:0/2371321979 <== mon.1 v1:172.21.15.179:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4209477549) 0x7ff2f00511a0 con 0x7ff2f8142690 2024-03-29T11:23:01.261 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:01.283+0000 7ff2f4ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:23:01.261 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:01.283+0000 7ff2f4ff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:23:01.261 INFO:teuthology.misc.health.smithi084.stdout: 2024-03-29T11:23:01.262 INFO:teuthology.misc.health.smithi084.stdout:{"epoch":10,"fsid":"2c3e002f-9eb0-4860-853b-4ef1ef562213","created":"2024-03-29T11:19:32.029169+0000","modified":"2024-03-29T11:19:34.200299+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:34.200299+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"5c39e276-2034-4716-bbc0-57a634eefa92","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"cd580e99-109f-4bf1-88c0-a5987bc8b9ed","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"11e60a5a-051d-453f-a190-66387cb35592","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"2c329fbe-bbc4-4030-bf87-d711cad04679","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f4213d4a-bd36-4433-91ae-d7e2b02481d3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"f0cf44ea-6055-48af-805b-aed858102ff9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"d895eaa0-cc0b-4ee4-9b5a-2dfcd89a60b7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"9bc5a7bc-8f52-4db9-96d6-de5e68d52eaa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:23:01.264 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:01.286+0000 7ff2fe739640 10 librados: watch_flush enter 2024-03-29T11:23:01.264 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:01.286+0000 7ff2fe739640 10 librados: watch_flush exit 2024-03-29T11:23:01.264 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:01.286+0000 7ff2fe739640 20 client.4333.objecter shutdown clearing up homeless session... 2024-03-29T11:23:01.264 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:01.286+0000 7ff2fe739640 10 client.4333.objecter successfully canceled tick 2024-03-29T11:23:01.264 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:01.286+0000 7ff2fe739640 1 -- 172.21.15.84:0/2371321979 >> v1:172.21.15.179:6816/2981134980 conn(0x7ff2d806dd60 legacy=0x7ff2d8070220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:01.264 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:01.286+0000 7ff2fe739640 10 monclient: shutdown 2024-03-29T11:23:01.264 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:01.286+0000 7ff2fe739640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:23:01.264 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:01.286+0000 7ff2fe739640 1 -- 172.21.15.84:0/2371321979 >> v1:172.21.15.179:6789/0 conn(0x7ff2f8142690 legacy=0x7ff2f812d530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:01.265 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:01.286+0000 7ff2fe739640 1 -- 172.21.15.84:0/2371321979 shutdown_connections 2024-03-29T11:23:01.265 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:01.286+0000 7ff2fe739640 1 -- 172.21.15.84:0/2371321979 >> 172.21.15.84:0/2371321979 conn(0x7ff2f8066a00 msgr2=0x7ff2f80b2490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:23:01.265 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:01.287+0000 7ff2fe739640 1 -- 172.21.15.84:0/2371321979 shutdown_connections 2024-03-29T11:23:01.265 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:01.287+0000 7ff2fe739640 1 -- 172.21.15.84:0/2371321979 wait complete. 2024-03-29T11:23:01.265 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:01.287+0000 7ff2fe739640 1 librados: shutdown 2024-03-29T11:23:01.278 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:23:08.085 DEBUG:teuthology.orchestra.run.smithi084:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:23:08.232 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:08.254+0000 7fddb3455640 10 monclient: get_monmap_and_config 2024-03-29T11:23:08.232 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:08.254+0000 7fddb3455640 10 monclient: build_initial_monmap 2024-03-29T11:23:08.232 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:08.254+0000 7fddb3455640 10 monclient: monmap: 2024-03-29T11:23:08.233 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-29T11:23:08.233 INFO:teuthology.misc.health.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:23:08.233 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T11:23:08.255055+0000 2024-03-29T11:23:08.233 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T11:23:08.255055+0000 2024-03-29T11:23:08.233 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-29T11:23:08.233 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:23:08.233 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-29T11:23:08.233 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.noname-c 2024-03-29T11:23:08.233 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-b 2024-03-29T11:23:08.233 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:23:08.234 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:08.256+0000 7fddb3455640 1 Processor -- start 2024-03-29T11:23:08.234 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:08.256+0000 7fddb3455640 1 -- start start 2024-03-29T11:23:08.234 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:08.256+0000 7fddb3455640 10 monclient: init 2024-03-29T11:23:08.236 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:08.257+0000 7fddb3455640 10 monclient: _reopen_session rank -1 2024-03-29T11:23:08.236 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:08.257+0000 7fddb3455640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:23:08.236 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:08.257+0000 7fddb3455640 10 monclient(hunting): picked mon.noname-b con 0x7fddac0b0500 addr v1:172.21.15.84:6790/0 2024-03-29T11:23:08.236 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:08.257+0000 7fddb3455640 10 monclient(hunting): picked mon.noname-c con 0x7fddac0b08e0 addr v1:172.21.15.179:6789/0 2024-03-29T11:23:08.236 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:08.257+0000 7fddb3455640 10 monclient(hunting): picked mon.noname-a con 0x7fddac0aeca0 addr v1:172.21.15.84:6789/0 2024-03-29T11:23:08.236 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:08.257+0000 7fddb3455640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fddac140620 con 0x7fddac0aeca0 2024-03-29T11:23:08.236 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:08.257+0000 7fddb3455640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fddac13daf0 con 0x7fddac0b08e0 2024-03-29T11:23:08.236 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:08.257+0000 7fddb3455640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fddac0ae180 con 0x7fddac0b0500 2024-03-29T11:23:08.236 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:08.257+0000 7fddb3455640 10 monclient(hunting): _renew_subs 2024-03-29T11:23:08.236 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:08.257+0000 7fddb3455640 10 monclient(hunting): authenticate will time out at 1049.551962s 2024-03-29T11:23:08.240 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:08.260+0000 7fddb09f0640 1 --1- >> v1:172.21.15.179:6789/0 conn(0x7fddac0b08e0 0x7fddac13d3e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.179:6789/0 says I am v1:172.21.15.84:56644/0 (socket says 172.21.15.84:56644) 2024-03-29T11:23:08.240 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:08.260+0000 7fddb09f0640 1 -- 172.21.15.84:0/387473621 learned_addr learned my addr 172.21.15.84:0/387473621 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-29T11:23:08.240 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:08.260+0000 7fddb19f2640 1 --1- >> v1:172.21.15.84:6789/0 conn(0x7fddac0aeca0 0x7fddac14b7d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.84:6789/0 says I am v1:172.21.15.84:36467/0 (socket says 172.21.15.84:36467) 2024-03-29T11:23:08.240 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:08.261+0000 7fdda3fff640 1 -- 172.21.15.84:0/387473621 <== mon.0 v1:172.21.15.84:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1885649228 0 0) 0x7fddac140620 con 0x7fddac0aeca0 2024-03-29T11:23:08.240 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:08.261+0000 7fdda3fff640 10 monclient(hunting): _negotiate 2024-03-29T11:23:08.240 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:08.261+0000 7fdda3fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:23:08.240 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:08.261+0000 7fdda3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:23:08.240 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:08.261+0000 7fdda3fff640 10 monclient(hunting): my global_id is 4374 2024-03-29T11:23:08.240 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:08.261+0000 7fdda3fff640 1 -- 172.21.15.84:0/387473621 --> v1:172.21.15.84:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdd98003520 con 0x7fddac0aeca0 2024-03-29T11:23:08.241 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:08.261+0000 7fdda3fff640 1 -- 172.21.15.84:0/387473621 <== mon.0 v1:172.21.15.84:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3854488635 0 0) 0x7fdd98003520 con 0x7fddac0aeca0 2024-03-29T11:23:08.241 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:08.261+0000 7fdda3fff640 1 -- 172.21.15.84:0/387473621 >> v1:172.21.15.84:6790/0 conn(0x7fddac0b0500 legacy=0x7fddac13ff10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:23:08.241 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:08.261+0000 7fdda3fff640 1 -- 172.21.15.84:0/387473621 >> v1:172.21.15.179:6789/0 conn(0x7fddac0b08e0 legacy=0x7fddac13d3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:08.241 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:08.261+0000 7fdda3fff640 10 monclient: _finish_hunting 0 2024-03-29T11:23:08.241 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:08.261+0000 7fdda3fff640 1 monclient: found mon.noname-a 2024-03-29T11:23:08.241 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:08.261+0000 7fdda3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:23:08.241 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:08.261+0000 7fdda3fff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.84:6789/0 2024-03-29T11:23:08.241 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:08.261+0000 7fdda3fff640 1 -- 172.21.15.84:0/387473621 --> v1:172.21.15.84:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fddac0accf0 con 0x7fddac0aeca0 2024-03-29T11:23:08.241 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:08.261+0000 7fdda3fff640 10 monclient: _finish_auth 0 2024-03-29T11:23:08.241 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:08.261+0000 7fdda3fff640 10 monclient: _check_auth_tickets 2024-03-29T11:23:08.241 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:08.261+0000 7fdda3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:23:08.241 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:08.261+0000 7fdda3fff640 1 -- 172.21.15.84:0/387473621 <== mon.0 v1:172.21.15.84:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4236868694 0 0) 0x7fdda8001690 con 0x7fddac0aeca0 2024-03-29T11:23:08.241 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:08.261+0000 7fdda3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:23:08.241 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:08.261+0000 7fdda3fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:23:08.241 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:08.261+0000 7fdda3fff640 10 monclient: dump: 2024-03-29T11:23:08.242 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T11:23:08.242 INFO:teuthology.misc.health.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:23:08.242 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T11:18:53.864025+0000 2024-03-29T11:23:08.242 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T11:18:53.864025+0000 2024-03-29T11:23:08.242 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T11:23:08.242 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:23:08.242 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T11:23:08.242 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T11:23:08.242 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T11:23:08.242 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:23:08.242 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:08.261+0000 7fddb3455640 5 monclient: authenticate success, global_id 4374 2024-03-29T11:23:08.242 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:08.261+0000 7fddb3455640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:23:08.242 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:08.262+0000 7fdda3fff640 1 -- 172.21.15.84:0/387473621 <== mon.0 v1:172.21.15.84:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fdda8001860 con 0x7fddac0aeca0 2024-03-29T11:23:08.242 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:08.262+0000 7fdda3fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:23:08.243 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:08.262+0000 7fdda3fff640 1 -- 172.21.15.84:0/387473621 <== mon.0 v1:172.21.15.84:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4236868694 0 0) 0x7fdda8001cb0 con 0x7fddac0aeca0 2024-03-29T11:23:08.243 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:08.262+0000 7fdda3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:23:08.243 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:08.262+0000 7fdda3fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:23:08.243 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:08.262+0000 7fdda3fff640 10 monclient: dump: 2024-03-29T11:23:08.243 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T11:23:08.243 INFO:teuthology.misc.health.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:23:08.243 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T11:18:53.864025+0000 2024-03-29T11:23:08.243 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T11:18:53.864025+0000 2024-03-29T11:23:08.243 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T11:23:08.243 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:23:08.243 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T11:23:08.243 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T11:23:08.243 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T11:23:08.243 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:23:08.244 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:08.262+0000 7fddb3455640 10 monclient: get_monmap_and_config success 2024-03-29T11:23:08.244 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:08.262+0000 7fddb3455640 4 set_mon_vals no callback set 2024-03-29T11:23:08.245 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:08.263+0000 7fddb3455640 10 monclient: shutdown 2024-03-29T11:23:08.245 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:08.263+0000 7fddb3455640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:23:08.245 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:08.263+0000 7fddb3455640 1 -- 172.21.15.84:0/387473621 >> v1:172.21.15.84:6789/0 conn(0x7fddac0aeca0 legacy=0x7fddac14b7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:08.245 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:08.263+0000 7fddb3455640 1 -- 172.21.15.84:0/387473621 shutdown_connections 2024-03-29T11:23:08.245 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:08.263+0000 7fddb3455640 1 -- 172.21.15.84:0/387473621 >> 172.21.15.84:0/387473621 conn(0x7fddac06b660 msgr2=0x7fddac06ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:23:08.245 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:08.264+0000 7fddb3455640 1 -- 172.21.15.84:0/387473621 shutdown_connections 2024-03-29T11:23:08.245 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:08.264+0000 7fddb3455640 1 -- 172.21.15.84:0/387473621 wait complete. 2024-03-29T11:23:08.246 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:08.265+0000 7fddb3455640 10 monclient: build_initial_monmap 2024-03-29T11:23:08.246 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:08.265+0000 7fddb3455640 10 monclient: monmap: 2024-03-29T11:23:08.246 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-29T11:23:08.246 INFO:teuthology.misc.health.smithi084.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:23:08.246 INFO:teuthology.misc.health.smithi084.stderr:last_changed 0.000000 2024-03-29T11:23:08.246 INFO:teuthology.misc.health.smithi084.stderr:created 0.000000 2024-03-29T11:23:08.246 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-29T11:23:08.246 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:23:08.246 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-29T11:23:08.246 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.noname-b 2024-03-29T11:23:08.246 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-c 2024-03-29T11:23:08.246 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:23:08.246 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:08.265+0000 7fddb3455640 1 librados: starting msgr at 2024-03-29T11:23:08.246 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:08.265+0000 7fddb3455640 1 librados: starting objecter 2024-03-29T11:23:08.247 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:08.266+0000 7fddb3455640 1 Processor -- start 2024-03-29T11:23:08.247 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:08.266+0000 7fddb3455640 1 -- start start 2024-03-29T11:23:08.247 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:08.266+0000 7fddb3455640 1 librados: setting wanted keys 2024-03-29T11:23:08.247 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:08.266+0000 7fddb3455640 1 librados: calling monclient init 2024-03-29T11:23:08.247 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:08.266+0000 7fddb3455640 10 monclient: init 2024-03-29T11:23:08.248 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:08.266+0000 7fddb3455640 10 monclient: _reopen_session rank -1 2024-03-29T11:23:08.248 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:08.266+0000 7fddb3455640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:23:08.248 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:08.266+0000 7fddb3455640 10 monclient(hunting): picked mon.noname-b con 0x7fddac0aeca0 addr v1:172.21.15.179:6789/0 2024-03-29T11:23:08.248 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:08.266+0000 7fddb3455640 10 monclient(hunting): picked mon.noname-a con 0x7fddac0b0500 addr v1:172.21.15.84:6789/0 2024-03-29T11:23:08.249 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:08.266+0000 7fddb3455640 10 monclient(hunting): picked mon.noname-c con 0x7fddac11ca80 addr v1:172.21.15.84:6790/0 2024-03-29T11:23:08.249 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:08.267+0000 7fddb3455640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fddac11dc40 con 0x7fddac0b0500 2024-03-29T11:23:08.249 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:08.267+0000 7fddb3455640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fddac11d720 con 0x7fddac0aeca0 2024-03-29T11:23:08.249 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:08.267+0000 7fddb3455640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fddac06ff00 con 0x7fddac11ca80 2024-03-29T11:23:08.249 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:08.267+0000 7fddb3455640 10 monclient(hunting): _renew_subs 2024-03-29T11:23:08.249 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:08.267+0000 7fddb3455640 10 monclient(hunting): authenticate will time out at 1049.561133s 2024-03-29T11:23:08.249 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:08.269+0000 7fddb11f1640 1 --1- >> v1:172.21.15.179:6789/0 conn(0x7fddac0aeca0 0x7fddac11cf30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.179:6789/0 says I am v1:172.21.15.84:60498/0 (socket says 172.21.15.84:60498) 2024-03-29T11:23:08.249 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:08.269+0000 7fddb11f1640 1 -- 172.21.15.84:0/1035464798 learned_addr learned my addr 172.21.15.84:0/1035464798 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-29T11:23:08.254 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:08.271+0000 7fdda1ffb640 10 client.?.objecter ms_handle_connect 0x7fddac0aeca0 2024-03-29T11:23:08.254 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:08.271+0000 7fdda1ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:23:08.254 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:08.271+0000 7fdda1ffb640 10 client.?.objecter ms_handle_connect 0x7fddac11ca80 2024-03-29T11:23:08.254 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:08.271+0000 7fdda1ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:23:08.254 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:08.271+0000 7fdda1ffb640 1 -- 172.21.15.84:0/1035464798 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4224990367 0 0) 0x7fddac06ff00 con 0x7fddac11ca80 2024-03-29T11:23:08.254 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:08.271+0000 7fdda1ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:23:08.254 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:08.271+0000 7fdda1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:23:08.254 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:08.271+0000 7fdda1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:23:08.254 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:08.271+0000 7fdda1ffb640 10 monclient(hunting): my global_id is 4367 2024-03-29T11:23:08.254 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:08.271+0000 7fdda1ffb640 1 -- 172.21.15.84:0/1035464798 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdd90004eb0 con 0x7fddac11ca80 2024-03-29T11:23:08.255 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:08.272+0000 7fdda1ffb640 10 client.?.objecter ms_handle_connect 0x7fddac0b0500 2024-03-29T11:23:08.255 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:08.272+0000 7fdda1ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:23:08.255 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:08.273+0000 7fdda1ffb640 1 -- 172.21.15.84:0/1035464798 <== mon.1 v1:172.21.15.179:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1645847881 0 0) 0x7fddac11d720 con 0x7fddac0aeca0 2024-03-29T11:23:08.255 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:08.273+0000 7fdda1ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:23:08.255 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:08.273+0000 7fdda1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:23:08.255 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:08.273+0000 7fdda1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:23:08.255 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:08.273+0000 7fdda1ffb640 10 monclient(hunting): my global_id is 4339 2024-03-29T11:23:08.255 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:08.273+0000 7fdda1ffb640 1 -- 172.21.15.84:0/1035464798 --> v1:172.21.15.179:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fddac06ff00 con 0x7fddac0aeca0 2024-03-29T11:23:08.255 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:08.273+0000 7fdda1ffb640 1 -- 172.21.15.84:0/1035464798 <== mon.0 v1:172.21.15.84:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 544928222 0 0) 0x7fddac11dc40 con 0x7fddac0b0500 2024-03-29T11:23:08.255 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:08.273+0000 7fdda1ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:23:08.255 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:08.273+0000 7fdda1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:23:08.255 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:08.273+0000 7fdda1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:23:08.255 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:08.273+0000 7fdda1ffb640 10 monclient(hunting): my global_id is 4377 2024-03-29T11:23:08.255 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:08.273+0000 7fdda1ffb640 1 -- 172.21.15.84:0/1035464798 --> v1:172.21.15.84:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fddac11d720 con 0x7fddac0b0500 2024-03-29T11:23:08.256 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:08.273+0000 7fdda1ffb640 1 -- 172.21.15.84:0/1035464798 <== mon.1 v1:172.21.15.179:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 704839867 0 0) 0x7fddac06ff00 con 0x7fddac0aeca0 2024-03-29T11:23:08.256 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:08.273+0000 7fdda1ffb640 1 -- 172.21.15.84:0/1035464798 --> v1:172.21.15.179:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fdd90005620 con 0x7fddac0aeca0 2024-03-29T11:23:08.256 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:08.274+0000 7fdda1ffb640 1 -- 172.21.15.84:0/1035464798 <== mon.1 v1:172.21.15.179:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4236868694 0 0) 0x7fdd9c004000 con 0x7fddac0aeca0 2024-03-29T11:23:08.256 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:08.274+0000 7fdda1ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:23:08.256 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:08.274+0000 7fdda1ffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:23:08.256 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:08.274+0000 7fdda1ffb640 10 monclient(hunting): dump: 2024-03-29T11:23:08.256 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T11:23:08.256 INFO:teuthology.misc.health.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:23:08.256 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T11:18:53.864025+0000 2024-03-29T11:23:08.256 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T11:18:53.864025+0000 2024-03-29T11:23:08.256 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T11:23:08.256 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:23:08.256 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T11:23:08.256 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T11:23:08.257 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T11:23:08.257 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:23:08.257 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:08.274+0000 7fdda1ffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:23:08.257 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:08.274+0000 7fdda1ffb640 1 -- 172.21.15.84:0/1035464798 <== mon.1 v1:172.21.15.179:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 631057937 0 0) 0x7fdd90005620 con 0x7fddac0aeca0 2024-03-29T11:23:08.257 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:08.274+0000 7fdda1ffb640 1 -- 172.21.15.84:0/1035464798 >> v1:172.21.15.84:6790/0 conn(0x7fddac11ca80 legacy=0x7fddac1d2c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:08.257 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:08.274+0000 7fdda1ffb640 1 -- 172.21.15.84:0/1035464798 >> v1:172.21.15.84:6789/0 conn(0x7fddac0b0500 legacy=0x7fddac13a930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:08.257 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:08.274+0000 7fdda1ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:23:08.257 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:08.274+0000 7fdda1ffb640 1 monclient: found mon.b 2024-03-29T11:23:08.257 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:08.274+0000 7fdda1ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:23:08.257 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:08.274+0000 7fdda1ffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-29T11:23:08.257 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:08.274+0000 7fdda1ffb640 1 -- 172.21.15.84:0/1035464798 --> v1:172.21.15.179:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fddac070090 con 0x7fddac0aeca0 2024-03-29T11:23:08.257 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:08.274+0000 7fdda1ffb640 10 monclient: _finish_auth 0 2024-03-29T11:23:08.257 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:08.274+0000 7fdda1ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:23:08.257 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:08.274+0000 7fdda1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:23:08.257 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:08.274+0000 7fddb3455640 5 monclient: authenticate success, global_id 4339 2024-03-29T11:23:08.258 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:08.274+0000 7fddb3455640 10 monclient: _renew_subs 2024-03-29T11:23:08.258 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:08.274+0000 7fddb3455640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-29T11:23:08.258 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:08.274+0000 7fddb3455640 1 -- 172.21.15.84:0/1035464798 --> v1:172.21.15.179:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fddac13b040 con 0x7fddac0aeca0 2024-03-29T11:23:08.258 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:08.274+0000 7fddb3455640 10 client.4339.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:23:08.258 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:08.274+0000 7fddb3455640 10 monclient: _renew_subs 2024-03-29T11:23:08.258 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:08.274+0000 7fddb3455640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-29T11:23:08.258 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:08.274+0000 7fddb3455640 1 -- 172.21.15.84:0/1035464798 --> v1:172.21.15.179:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fddac1d3390 con 0x7fddac0aeca0 2024-03-29T11:23:08.258 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:08.274+0000 7fddb3455640 1 librados: init done 2024-03-29T11:23:08.258 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:08.275+0000 7fdda1ffb640 1 -- 172.21.15.84:0/1035464798 <== mon.1 v1:172.21.15.179:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fdd9c000ed0 con 0x7fddac0aeca0 2024-03-29T11:23:08.258 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:08.275+0000 7fdda1ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:23:08.258 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:08.275+0000 7fdda37fe640 4 set_mon_vals no callback set 2024-03-29T11:23:08.258 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:08.275+0000 7fdda1ffb640 1 -- 172.21.15.84:0/1035464798 <== mon.1 v1:172.21.15.179:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4236868694 0 0) 0x7fdd9c0012a0 con 0x7fddac0aeca0 2024-03-29T11:23:08.258 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:08.275+0000 7fdda1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:23:08.259 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:08.275+0000 7fdda1ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:23:08.259 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:08.275+0000 7fdda1ffb640 10 monclient: dump: 2024-03-29T11:23:08.259 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T11:23:08.259 INFO:teuthology.misc.health.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:23:08.259 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T11:18:53.864025+0000 2024-03-29T11:23:08.259 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T11:18:53.864025+0000 2024-03-29T11:23:08.259 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T11:23:08.259 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:23:08.259 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T11:23:08.259 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T11:23:08.259 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T11:23:08.259 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:23:08.259 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:08.276+0000 7fddb3455640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:23:08.260 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:08.276+0000 7fddb3455640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:23:08.260 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:08.276+0000 7fddb3455640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-29T11:23:08.260 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:08.276+0000 7fddb3455640 1 -- 172.21.15.84:0/1035464798 --> v1:172.21.15.179:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdd8c005080 con 0x7fddac0aeca0 2024-03-29T11:23:08.261 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:08.277+0000 7fdda1ffb640 1 -- 172.21.15.84:0/1035464798 <== mon.1 v1:172.21.15.179:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4033942909 0 0) 0x7fdd9c001400 con 0x7fddac0aeca0 2024-03-29T11:23:08.261 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:08.278+0000 7fdda1ffb640 1 -- 172.21.15.84:0/1035464798 <== mon.1 v1:172.21.15.179:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2808139587 0 0) 0x7fdd9c084970 con 0x7fddac0aeca0 2024-03-29T11:23:08.261 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:08.278+0000 7fdda1ffb640 10 client.4339.objecter ms_dispatch 0x7fddac000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:23:08.261 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:08.278+0000 7fdda1ffb640 3 client.4339.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:23:08.261 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:08.278+0000 7fdda1ffb640 3 client.4339.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:23:08.261 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:08.278+0000 7fdda1ffb640 20 client.4339.objecter dump_active .. 0 homeless 2024-03-29T11:23:08.261 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:08.279+0000 7fddb09f0640 10 monclient: get_auth_request con 0x7fdd9006dff0 auth_method 0 2024-03-29T11:23:08.261 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:08.281+0000 7fdda1ffb640 10 client.4339.objecter ms_handle_connect 0x7fdd9006dff0 2024-03-29T11:23:08.263 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:08.285+0000 7fdda1ffb640 1 -- 172.21.15.84:0/1035464798 <== mon.1 v1:172.21.15.179:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fdd9c050250 con 0x7fddac0aeca0 2024-03-29T11:23:08.263 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:08.285+0000 7fdda1ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:23:08.263 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:08.285+0000 7fdda1ffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:23:08.590 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:08.611+0000 7fddb3455640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:23:08.590 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:08.612+0000 7fddb3455640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:23:08.590 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:08.612+0000 7fddb3455640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-29T11:23:08.590 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:08.612+0000 7fddb3455640 1 -- 172.21.15.84:0/1035464798 --> v1:172.21.15.179:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fdd8c005330 con 0x7fddac0aeca0 2024-03-29T11:23:08.591 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:08.613+0000 7fdda1ffb640 1 -- 172.21.15.84:0/1035464798 <== mon.1 v1:172.21.15.179:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4209477549) 0x7fdd9c052180 con 0x7fddac0aeca0 2024-03-29T11:23:08.591 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:08.613+0000 7fdda1ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:23:08.591 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:08.613+0000 7fdda1ffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:23:08.591 INFO:teuthology.misc.health.smithi084.stdout: 2024-03-29T11:23:08.591 INFO:teuthology.misc.health.smithi084.stdout:{"epoch":10,"fsid":"2c3e002f-9eb0-4860-853b-4ef1ef562213","created":"2024-03-29T11:19:32.029169+0000","modified":"2024-03-29T11:19:34.200299+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:34.200299+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"5c39e276-2034-4716-bbc0-57a634eefa92","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"cd580e99-109f-4bf1-88c0-a5987bc8b9ed","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"11e60a5a-051d-453f-a190-66387cb35592","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"2c329fbe-bbc4-4030-bf87-d711cad04679","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f4213d4a-bd36-4433-91ae-d7e2b02481d3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"f0cf44ea-6055-48af-805b-aed858102ff9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"d895eaa0-cc0b-4ee4-9b5a-2dfcd89a60b7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"9bc5a7bc-8f52-4db9-96d6-de5e68d52eaa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:23:08.595 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:08.616+0000 7fddb3455640 10 librados: watch_flush enter 2024-03-29T11:23:08.595 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:08.616+0000 7fddb3455640 10 librados: watch_flush exit 2024-03-29T11:23:08.595 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:08.616+0000 7fddb3455640 20 client.4339.objecter shutdown clearing up homeless session... 2024-03-29T11:23:08.595 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:08.616+0000 7fddb3455640 10 client.4339.objecter successfully canceled tick 2024-03-29T11:23:08.595 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:08.617+0000 7fddb3455640 1 -- 172.21.15.84:0/1035464798 >> v1:172.21.15.179:6816/2981134980 conn(0x7fdd9006dff0 legacy=0x7fdd900704b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:08.595 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:08.617+0000 7fddb3455640 10 monclient: shutdown 2024-03-29T11:23:08.596 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:08.617+0000 7fddb3455640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:23:08.596 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:08.617+0000 7fddb3455640 1 -- 172.21.15.84:0/1035464798 >> v1:172.21.15.179:6789/0 conn(0x7fddac0aeca0 legacy=0x7fddac11cf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:08.596 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:08.617+0000 7fddb3455640 1 -- 172.21.15.84:0/1035464798 shutdown_connections 2024-03-29T11:23:08.596 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:08.617+0000 7fddb3455640 1 -- 172.21.15.84:0/1035464798 >> 172.21.15.84:0/1035464798 conn(0x7fddac06b660 msgr2=0x7fddac0a9c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:23:08.596 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:08.618+0000 7fddb3455640 1 -- 172.21.15.84:0/1035464798 shutdown_connections 2024-03-29T11:23:08.597 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:08.618+0000 7fddb3455640 1 -- 172.21.15.84:0/1035464798 wait complete. 2024-03-29T11:23:08.597 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:08.618+0000 7fddb3455640 1 librados: shutdown 2024-03-29T11:23:08.613 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:23:15.418 DEBUG:teuthology.orchestra.run.smithi084:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:23:15.561 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:15.583+0000 7fbc9a1e9640 10 monclient: get_monmap_and_config 2024-03-29T11:23:15.562 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:15.583+0000 7fbc9a1e9640 10 monclient: build_initial_monmap 2024-03-29T11:23:15.562 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:15.583+0000 7fbc9a1e9640 10 monclient: monmap: 2024-03-29T11:23:15.562 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-29T11:23:15.562 INFO:teuthology.misc.health.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:23:15.562 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T11:23:15.584135+0000 2024-03-29T11:23:15.562 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T11:23:15.584135+0000 2024-03-29T11:23:15.562 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-29T11:23:15.562 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:23:15.562 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-29T11:23:15.562 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.noname-c 2024-03-29T11:23:15.562 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-b 2024-03-29T11:23:15.562 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:23:15.564 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:15.585+0000 7fbc9a1e9640 1 Processor -- start 2024-03-29T11:23:15.564 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:15.585+0000 7fbc9a1e9640 1 -- start start 2024-03-29T11:23:15.564 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:15.585+0000 7fbc9a1e9640 10 monclient: init 2024-03-29T11:23:15.565 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:15.586+0000 7fbc9a1e9640 10 monclient: _reopen_session rank -1 2024-03-29T11:23:15.565 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:15.586+0000 7fbc9a1e9640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:23:15.565 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:15.586+0000 7fbc9a1e9640 10 monclient(hunting): picked mon.noname-b con 0x7fbc9406d9d0 addr v1:172.21.15.84:6790/0 2024-03-29T11:23:15.566 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:15.586+0000 7fbc9a1e9640 10 monclient(hunting): picked mon.noname-c con 0x7fbc9413ef30 addr v1:172.21.15.179:6789/0 2024-03-29T11:23:15.566 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:15.586+0000 7fbc9a1e9640 10 monclient(hunting): picked mon.noname-a con 0x7fbc940b0860 addr v1:172.21.15.84:6789/0 2024-03-29T11:23:15.566 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:15.586+0000 7fbc9a1e9640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbc940aed10 con 0x7fbc940b0860 2024-03-29T11:23:15.566 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:15.586+0000 7fbc9a1e9640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbc940ab1b0 con 0x7fbc9413ef30 2024-03-29T11:23:15.566 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:15.586+0000 7fbc9a1e9640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbc940ab490 con 0x7fbc9406d9d0 2024-03-29T11:23:15.566 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:15.586+0000 7fbc9a1e9640 10 monclient(hunting): _renew_subs 2024-03-29T11:23:15.566 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:15.586+0000 7fbc9a1e9640 10 monclient(hunting): authenticate will time out at 1056.880856s 2024-03-29T11:23:15.566 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:15.586+0000 7fbc937fe640 1 --1- >> v1:172.21.15.84:6790/0 conn(0x7fbc9406d9d0 0x7fbc9413e820 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.84:6790/0 says I am v1:172.21.15.84:43146/0 (socket says 172.21.15.84:43146) 2024-03-29T11:23:15.566 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:15.586+0000 7fbc937fe640 1 -- 172.21.15.84:0/1486269539 learned_addr learned my addr 172.21.15.84:0/1486269539 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-29T11:23:15.566 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:15.587+0000 7fbc927fc640 1 -- 172.21.15.84:0/1486269539 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 643163217 0 0) 0x7fbc940ab490 con 0x7fbc9406d9d0 2024-03-29T11:23:15.566 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:15.587+0000 7fbc927fc640 10 monclient(hunting): _negotiate 2024-03-29T11:23:15.566 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:15.587+0000 7fbc927fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:23:15.566 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:15.587+0000 7fbc927fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:23:15.566 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:15.587+0000 7fbc927fc640 10 monclient(hunting): my global_id is 4370 2024-03-29T11:23:15.566 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:15.587+0000 7fbc927fc640 1 -- 172.21.15.84:0/1486269539 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbc7c003520 con 0x7fbc9406d9d0 2024-03-29T11:23:15.567 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:15.588+0000 7fbc927fc640 1 -- 172.21.15.84:0/1486269539 <== mon.2 v1:172.21.15.84:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3566387091 0 0) 0x7fbc7c003520 con 0x7fbc9406d9d0 2024-03-29T11:23:15.567 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:15.588+0000 7fbc927fc640 1 -- 172.21.15.84:0/1486269539 >> v1:172.21.15.179:6789/0 conn(0x7fbc9413ef30 legacy=0x7fbc940af1d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:23:15.569 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:15.591+0000 7fbc927fc640 1 -- 172.21.15.84:0/1486269539 >> v1:172.21.15.84:6789/0 conn(0x7fbc940b0860 legacy=0x7fbc94142a30 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:23:15.573 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:15.593+0000 7fbc927fc640 10 monclient: _finish_hunting 0 2024-03-29T11:23:15.573 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:15.593+0000 7fbc927fc640 1 monclient: found mon.noname-b 2024-03-29T11:23:15.573 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:15.593+0000 7fbc927fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:23:15.574 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:15.593+0000 7fbc927fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.84:6790/0 2024-03-29T11:23:15.574 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:15.593+0000 7fbc927fc640 1 -- 172.21.15.84:0/1486269539 --> v1:172.21.15.84:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc940ab770 con 0x7fbc9406d9d0 2024-03-29T11:23:15.574 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:15.593+0000 7fbc927fc640 10 monclient: _finish_auth 0 2024-03-29T11:23:15.574 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:15.593+0000 7fbc927fc640 10 monclient: _check_auth_tickets 2024-03-29T11:23:15.574 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:15.593+0000 7fbc927fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:23:15.574 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:15.593+0000 7fbc9a1e9640 5 monclient: authenticate success, global_id 4370 2024-03-29T11:23:15.574 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:15.593+0000 7fbc9a1e9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:23:15.574 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:15.593+0000 7fbc927fc640 1 -- 172.21.15.84:0/1486269539 <== mon.2 v1:172.21.15.84:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4236868694 0 0) 0x7fbc8c001a10 con 0x7fbc9406d9d0 2024-03-29T11:23:15.574 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:15.593+0000 7fbc927fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:23:15.574 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:15.593+0000 7fbc927fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:23:15.574 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:15.593+0000 7fbc927fc640 10 monclient: dump: 2024-03-29T11:23:15.574 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T11:23:15.574 INFO:teuthology.misc.health.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:23:15.574 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T11:18:53.864025+0000 2024-03-29T11:23:15.575 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T11:18:53.864025+0000 2024-03-29T11:23:15.575 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T11:23:15.575 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:23:15.575 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T11:23:15.575 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T11:23:15.575 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T11:23:15.575 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:23:15.575 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:15.593+0000 7fbc9a1e9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:23:15.575 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:15.593+0000 7fbc927fc640 1 -- 172.21.15.84:0/1486269539 <== mon.2 v1:172.21.15.84:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbc8c001ca0 con 0x7fbc9406d9d0 2024-03-29T11:23:15.575 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:15.593+0000 7fbc927fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:23:15.575 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:15.594+0000 7fbc9a1e9640 10 monclient: get_monmap_and_config success 2024-03-29T11:23:15.575 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:15.594+0000 7fbc9a1e9640 4 set_mon_vals no callback set 2024-03-29T11:23:15.575 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:15.594+0000 7fbc9a1e9640 10 monclient: shutdown 2024-03-29T11:23:15.575 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:15.594+0000 7fbc9a1e9640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:23:15.575 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:15.594+0000 7fbc9a1e9640 1 -- 172.21.15.84:0/1486269539 >> v1:172.21.15.84:6790/0 conn(0x7fbc9406d9d0 legacy=0x7fbc9413e820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:15.576 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:15.595+0000 7fbc9a1e9640 1 -- 172.21.15.84:0/1486269539 shutdown_connections 2024-03-29T11:23:15.576 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:15.595+0000 7fbc9a1e9640 1 -- 172.21.15.84:0/1486269539 >> 172.21.15.84:0/1486269539 conn(0x7fbc9406b660 msgr2=0x7fbc9406ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:23:15.576 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:15.597+0000 7fbc9a1e9640 1 -- 172.21.15.84:0/1486269539 shutdown_connections 2024-03-29T11:23:15.576 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:15.597+0000 7fbc9a1e9640 1 -- 172.21.15.84:0/1486269539 wait complete. 2024-03-29T11:23:15.577 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:15.598+0000 7fbc9a1e9640 10 monclient: build_initial_monmap 2024-03-29T11:23:15.577 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:15.598+0000 7fbc9a1e9640 10 monclient: monmap: 2024-03-29T11:23:15.577 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-29T11:23:15.577 INFO:teuthology.misc.health.smithi084.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:23:15.577 INFO:teuthology.misc.health.smithi084.stderr:last_changed 0.000000 2024-03-29T11:23:15.577 INFO:teuthology.misc.health.smithi084.stderr:created 0.000000 2024-03-29T11:23:15.577 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-29T11:23:15.577 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:23:15.578 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-29T11:23:15.578 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.noname-b 2024-03-29T11:23:15.578 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-c 2024-03-29T11:23:15.578 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:23:15.578 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:15.599+0000 7fbc9a1e9640 1 librados: starting msgr at 2024-03-29T11:23:15.578 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:15.599+0000 7fbc9a1e9640 1 librados: starting objecter 2024-03-29T11:23:15.578 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:15.599+0000 7fbc9a1e9640 1 Processor -- start 2024-03-29T11:23:15.584 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:15.599+0000 7fbc9a1e9640 1 -- start start 2024-03-29T11:23:15.584 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:15.599+0000 7fbc9a1e9640 1 librados: setting wanted keys 2024-03-29T11:23:15.585 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:15.599+0000 7fbc9a1e9640 1 librados: calling monclient init 2024-03-29T11:23:15.585 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:15.599+0000 7fbc9a1e9640 10 monclient: init 2024-03-29T11:23:15.585 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:15.600+0000 7fbc9a1e9640 10 monclient: _reopen_session rank -1 2024-03-29T11:23:15.585 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:15.600+0000 7fbc9a1e9640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:23:15.585 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:15.600+0000 7fbc9a1e9640 10 monclient(hunting): picked mon.noname-c con 0x7fbc9406d9d0 addr v1:172.21.15.84:6790/0 2024-03-29T11:23:15.585 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:15.600+0000 7fbc9a1e9640 10 monclient(hunting): picked mon.noname-b con 0x7fbc940b0860 addr v1:172.21.15.179:6789/0 2024-03-29T11:23:15.585 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:15.600+0000 7fbc9a1e9640 10 monclient(hunting): picked mon.noname-a con 0x7fbc9413ef30 addr v1:172.21.15.84:6789/0 2024-03-29T11:23:15.585 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:15.600+0000 7fbc9a1e9640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbc9414fc30 con 0x7fbc9413ef30 2024-03-29T11:23:15.585 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:15.600+0000 7fbc9a1e9640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbc940700f0 con 0x7fbc940b0860 2024-03-29T11:23:15.585 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:15.600+0000 7fbc9a1e9640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbc9406f250 con 0x7fbc9406d9d0 2024-03-29T11:23:15.585 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:15.600+0000 7fbc9a1e9640 10 monclient(hunting): _renew_subs 2024-03-29T11:23:15.585 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:15.600+0000 7fbc9a1e9640 10 monclient(hunting): authenticate will time out at 1056.894478s 2024-03-29T11:23:15.585 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:15.600+0000 7fbc937fe640 1 --1- >> v1:172.21.15.84:6790/0 conn(0x7fbc9406d9d0 0x7fbc94070b60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.84:6790/0 says I am v1:172.21.15.84:46990/0 (socket says 172.21.15.84:46990) 2024-03-29T11:23:15.586 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:15.600+0000 7fbc937fe640 1 -- 172.21.15.84:0/705889229 learned_addr learned my addr 172.21.15.84:0/705889229 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-29T11:23:15.586 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:15.601+0000 7fbc7bfff640 10 client.?.objecter ms_handle_connect 0x7fbc9406d9d0 2024-03-29T11:23:15.586 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:15.601+0000 7fbc7bfff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:23:15.586 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:15.601+0000 7fbc7bfff640 1 -- 172.21.15.84:0/705889229 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4059999336 0 0) 0x7fbc9406f250 con 0x7fbc9406d9d0 2024-03-29T11:23:15.586 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:15.601+0000 7fbc7bfff640 10 monclient(hunting): _negotiate 2024-03-29T11:23:15.586 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:15.601+0000 7fbc7bfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:23:15.586 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:15.601+0000 7fbc7bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:23:15.586 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:15.601+0000 7fbc7bfff640 10 monclient(hunting): my global_id is 4373 2024-03-29T11:23:15.586 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:15.601+0000 7fbc7bfff640 1 -- 172.21.15.84:0/705889229 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbc80004eb0 con 0x7fbc9406d9d0 2024-03-29T11:23:15.586 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:15.601+0000 7fbc7bfff640 1 -- 172.21.15.84:0/705889229 <== mon.2 v1:172.21.15.84:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2535383435 0 0) 0x7fbc80004eb0 con 0x7fbc9406d9d0 2024-03-29T11:23:15.586 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:15.601+0000 7fbc7bfff640 1 -- 172.21.15.84:0/705889229 --> v1:172.21.15.84:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbc9406f250 con 0x7fbc9406d9d0 2024-03-29T11:23:15.586 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:15.601+0000 7fbc7bfff640 1 -- 172.21.15.84:0/705889229 <== mon.2 v1:172.21.15.84:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4236868694 0 0) 0x7fbc8c004e60 con 0x7fbc9406d9d0 2024-03-29T11:23:15.586 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:15.601+0000 7fbc7bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:23:15.586 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:15.601+0000 7fbc7bfff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:23:15.587 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:15.601+0000 7fbc7bfff640 10 monclient(hunting): dump: 2024-03-29T11:23:15.587 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T11:23:15.587 INFO:teuthology.misc.health.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:23:15.587 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T11:18:53.864025+0000 2024-03-29T11:23:15.587 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T11:18:53.864025+0000 2024-03-29T11:23:15.587 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T11:23:15.587 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:23:15.587 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T11:23:15.587 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T11:23:15.587 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T11:23:15.587 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:23:15.587 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:15.601+0000 7fbc7bfff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:23:15.587 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:15.601+0000 7fbc7bfff640 1 -- 172.21.15.84:0/705889229 <== mon.2 v1:172.21.15.84:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3568643840 0 0) 0x7fbc9406f250 con 0x7fbc9406d9d0 2024-03-29T11:23:15.587 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:15.601+0000 7fbc7bfff640 1 -- 172.21.15.84:0/705889229 >> v1:172.21.15.179:6789/0 conn(0x7fbc940b0860 legacy=0x7fbc9414ee10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:23:15.587 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:15.601+0000 7fbc7bfff640 1 -- 172.21.15.84:0/705889229 >> v1:172.21.15.84:6789/0 conn(0x7fbc9413ef30 legacy=0x7fbc9414f520 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:23:15.588 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:15.601+0000 7fbc7bfff640 10 monclient: _finish_hunting 0 2024-03-29T11:23:15.588 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:15.601+0000 7fbc7bfff640 1 monclient: found mon.c 2024-03-29T11:23:15.588 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:15.601+0000 7fbc7bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:23:15.588 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:15.601+0000 7fbc7bfff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-29T11:23:15.588 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:15.601+0000 7fbc7bfff640 1 -- 172.21.15.84:0/705889229 --> v1:172.21.15.84:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc9406e830 con 0x7fbc9406d9d0 2024-03-29T11:23:15.588 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:15.601+0000 7fbc7bfff640 10 monclient: _finish_auth 0 2024-03-29T11:23:15.588 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:15.601+0000 7fbc7bfff640 10 monclient: _check_auth_tickets 2024-03-29T11:23:15.588 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:15.601+0000 7fbc7bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:23:15.588 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:15.602+0000 7fbc9a1e9640 5 monclient: authenticate success, global_id 4373 2024-03-29T11:23:15.588 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:15.602+0000 7fbc9a1e9640 10 monclient: _renew_subs 2024-03-29T11:23:15.588 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:15.602+0000 7fbc9a1e9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-29T11:23:15.588 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:15.602+0000 7fbc9a1e9640 1 -- 172.21.15.84:0/705889229 --> v1:172.21.15.84:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbc9406ea80 con 0x7fbc9406d9d0 2024-03-29T11:23:15.588 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:15.602+0000 7fbc9a1e9640 10 client.4373.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:23:15.588 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:15.602+0000 7fbc9a1e9640 10 monclient: _renew_subs 2024-03-29T11:23:15.588 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:15.602+0000 7fbc9a1e9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-29T11:23:15.588 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:15.602+0000 7fbc9a1e9640 1 -- 172.21.15.84:0/705889229 --> v1:172.21.15.84:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fbc940b1d60 con 0x7fbc9406d9d0 2024-03-29T11:23:15.589 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:15.602+0000 7fbc9a1e9640 1 librados: init done 2024-03-29T11:23:15.589 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:15.602+0000 7fbc7bfff640 1 -- 172.21.15.84:0/705889229 <== mon.2 v1:172.21.15.84:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbc8c0052e0 con 0x7fbc9406d9d0 2024-03-29T11:23:15.589 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:15.602+0000 7fbc7bfff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:23:15.589 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:15.602+0000 7fbc7bfff640 1 -- 172.21.15.84:0/705889229 <== mon.2 v1:172.21.15.84:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4236868694 0 0) 0x7fbc8c0056f0 con 0x7fbc9406d9d0 2024-03-29T11:23:15.589 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:15.602+0000 7fbc7bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:23:15.589 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:15.602+0000 7fbc7bfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:23:15.589 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:15.602+0000 7fbc7bfff640 10 monclient: dump: 2024-03-29T11:23:15.589 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T11:23:15.589 INFO:teuthology.misc.health.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:23:15.589 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T11:18:53.864025+0000 2024-03-29T11:23:15.589 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T11:18:53.864025+0000 2024-03-29T11:23:15.589 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T11:23:15.589 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:23:15.589 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T11:23:15.589 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T11:23:15.589 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T11:23:15.589 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:23:15.590 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:15.603+0000 7fbc91ffb640 4 set_mon_vals no callback set 2024-03-29T11:23:15.590 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:15.603+0000 7fbc9a1e9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:23:15.590 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:15.604+0000 7fbc9a1e9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:23:15.590 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:15.604+0000 7fbc9a1e9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-29T11:23:15.590 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:15.604+0000 7fbc9a1e9640 1 -- 172.21.15.84:0/705889229 --> v1:172.21.15.84:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbc74005080 con 0x7fbc9406d9d0 2024-03-29T11:23:15.591 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:15.607+0000 7fbc7bfff640 1 -- 172.21.15.84:0/705889229 <== mon.2 v1:172.21.15.84:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4033942909 0 0) 0x7fbc8c001e90 con 0x7fbc9406d9d0 2024-03-29T11:23:15.591 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:15.607+0000 7fbc7bfff640 1 -- 172.21.15.84:0/705889229 <== mon.2 v1:172.21.15.84:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2808139587 0 0) 0x7fbc8c0851c0 con 0x7fbc9406d9d0 2024-03-29T11:23:15.591 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:15.607+0000 7fbc7bfff640 10 client.4373.objecter ms_dispatch 0x7fbc94000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:23:15.591 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:15.607+0000 7fbc7bfff640 3 client.4373.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:23:15.591 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:15.607+0000 7fbc7bfff640 3 client.4373.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:23:15.591 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:15.607+0000 7fbc7bfff640 20 client.4373.objecter dump_active .. 0 homeless 2024-03-29T11:23:15.591 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:15.608+0000 7fbc92ffd640 10 monclient: get_auth_request con 0x7fbc8006dbb0 auth_method 0 2024-03-29T11:23:15.592 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:15.608+0000 7fbc7bfff640 10 client.4373.objecter ms_handle_connect 0x7fbc8006dbb0 2024-03-29T11:23:15.595 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:15.616+0000 7fbc7bfff640 1 -- 172.21.15.84:0/705889229 <== mon.2 v1:172.21.15.84:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fbc8c0508d0 con 0x7fbc9406d9d0 2024-03-29T11:23:15.595 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:15.617+0000 7fbc7bfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:23:15.595 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:15.617+0000 7fbc7bfff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:23:15.921 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:15.942+0000 7fbc9a1e9640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:23:15.921 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:15.942+0000 7fbc9a1e9640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:23:15.921 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:15.942+0000 7fbc9a1e9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-29T11:23:15.921 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:15.942+0000 7fbc9a1e9640 1 -- 172.21.15.84:0/705889229 --> v1:172.21.15.84:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fbc74005330 con 0x7fbc9406d9d0 2024-03-29T11:23:15.922 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:15.943+0000 7fbc7bfff640 1 -- 172.21.15.84:0/705889229 <== mon.2 v1:172.21.15.84:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4209477549) 0x7fbc8c0529d0 con 0x7fbc9406d9d0 2024-03-29T11:23:15.922 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:15.943+0000 7fbc7bfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:23:15.922 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:15.943+0000 7fbc7bfff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:23:15.922 INFO:teuthology.misc.health.smithi084.stdout: 2024-03-29T11:23:15.923 INFO:teuthology.misc.health.smithi084.stdout:{"epoch":10,"fsid":"2c3e002f-9eb0-4860-853b-4ef1ef562213","created":"2024-03-29T11:19:32.029169+0000","modified":"2024-03-29T11:19:34.200299+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:34.200299+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"5c39e276-2034-4716-bbc0-57a634eefa92","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"cd580e99-109f-4bf1-88c0-a5987bc8b9ed","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"11e60a5a-051d-453f-a190-66387cb35592","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"2c329fbe-bbc4-4030-bf87-d711cad04679","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f4213d4a-bd36-4433-91ae-d7e2b02481d3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"f0cf44ea-6055-48af-805b-aed858102ff9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"d895eaa0-cc0b-4ee4-9b5a-2dfcd89a60b7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"9bc5a7bc-8f52-4db9-96d6-de5e68d52eaa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:23:15.924 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:15.946+0000 7fbc9a1e9640 10 librados: watch_flush enter 2024-03-29T11:23:15.924 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:15.946+0000 7fbc9a1e9640 10 librados: watch_flush exit 2024-03-29T11:23:15.924 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:15.946+0000 7fbc9a1e9640 20 client.4373.objecter shutdown clearing up homeless session... 2024-03-29T11:23:15.924 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:15.946+0000 7fbc9a1e9640 10 client.4373.objecter successfully canceled tick 2024-03-29T11:23:15.925 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:15.947+0000 7fbc9a1e9640 1 -- 172.21.15.84:0/705889229 >> v1:172.21.15.179:6816/2981134980 conn(0x7fbc8006dbb0 legacy=0x7fbc80070070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:15.925 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:15.947+0000 7fbc9a1e9640 10 monclient: shutdown 2024-03-29T11:23:15.925 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:15.947+0000 7fbc9a1e9640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:23:15.925 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:15.947+0000 7fbc9a1e9640 1 -- 172.21.15.84:0/705889229 >> v1:172.21.15.84:6790/0 conn(0x7fbc9406d9d0 legacy=0x7fbc94070b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:15.926 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:15.948+0000 7fbc9a1e9640 1 -- 172.21.15.84:0/705889229 shutdown_connections 2024-03-29T11:23:15.926 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:15.948+0000 7fbc9a1e9640 1 -- 172.21.15.84:0/705889229 >> 172.21.15.84:0/705889229 conn(0x7fbc9406b660 msgr2=0x7fbc9413d4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:23:15.927 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:15.948+0000 7fbc9a1e9640 1 -- 172.21.15.84:0/705889229 shutdown_connections 2024-03-29T11:23:15.927 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:15.948+0000 7fbc9a1e9640 1 -- 172.21.15.84:0/705889229 wait complete. 2024-03-29T11:23:15.927 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:15.949+0000 7fbc9a1e9640 1 librados: shutdown 2024-03-29T11:23:15.945 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:23:22.751 DEBUG:teuthology.orchestra.run.smithi084:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:23:22.898 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:22.919+0000 7febce458640 10 monclient: get_monmap_and_config 2024-03-29T11:23:22.898 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:22.919+0000 7febce458640 10 monclient: build_initial_monmap 2024-03-29T11:23:22.898 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:22.919+0000 7febce458640 10 monclient: monmap: 2024-03-29T11:23:22.898 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-29T11:23:22.898 INFO:teuthology.misc.health.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:23:22.898 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T11:23:22.920359+0000 2024-03-29T11:23:22.898 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T11:23:22.920359+0000 2024-03-29T11:23:22.898 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-29T11:23:22.898 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:23:22.899 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-29T11:23:22.899 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.noname-c 2024-03-29T11:23:22.899 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-b 2024-03-29T11:23:22.899 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:23:22.901 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:22.922+0000 7febce458640 1 Processor -- start 2024-03-29T11:23:22.902 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:22.923+0000 7febce458640 1 -- start start 2024-03-29T11:23:22.903 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:22.923+0000 7febce458640 10 monclient: init 2024-03-29T11:23:22.903 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:22.923+0000 7febce458640 10 monclient: _reopen_session rank -1 2024-03-29T11:23:22.903 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:22.923+0000 7febce458640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:23:22.903 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:22.923+0000 7febce458640 10 monclient(hunting): picked mon.noname-c con 0x7febc80b2800 addr v1:172.21.15.179:6789/0 2024-03-29T11:23:22.903 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:22.923+0000 7febce458640 10 monclient(hunting): picked mon.noname-b con 0x7febc80a9c00 addr v1:172.21.15.84:6790/0 2024-03-29T11:23:22.903 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:22.923+0000 7febce458640 10 monclient(hunting): picked mon.noname-a con 0x7febc80b38c0 addr v1:172.21.15.84:6789/0 2024-03-29T11:23:22.903 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:22.923+0000 7febce458640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7febc8141eb0 con 0x7febc80b38c0 2024-03-29T11:23:22.903 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:22.923+0000 7febce458640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7febc80b22d0 con 0x7febc80b2800 2024-03-29T11:23:22.903 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:22.923+0000 7febce458640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7febc8142d30 con 0x7febc80a9c00 2024-03-29T11:23:22.903 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:22.923+0000 7febce458640 10 monclient(hunting): _renew_subs 2024-03-29T11:23:22.903 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:22.923+0000 7febce458640 10 monclient(hunting): authenticate will time out at 1064.218078s 2024-03-29T11:23:22.910 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:22.928+0000 7febcdc57640 1 --1- >> v1:172.21.15.84:6789/0 conn(0x7febc80b38c0 0x7febc814d7d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.84:6789/0 says I am v1:172.21.15.84:40571/0 (socket says 172.21.15.84:40571) 2024-03-29T11:23:22.910 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:22.930+0000 7febcdc57640 1 -- 172.21.15.84:0/3660444962 learned_addr learned my addr 172.21.15.84:0/3660444962 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-29T11:23:22.910 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:22.931+0000 7febbffff640 1 -- 172.21.15.84:0/3660444962 <== mon.0 v1:172.21.15.84:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2943546904 0 0) 0x7febc8141eb0 con 0x7febc80b38c0 2024-03-29T11:23:22.910 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:22.931+0000 7febbffff640 10 monclient(hunting): _negotiate 2024-03-29T11:23:22.910 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:22.931+0000 7febbffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:23:22.910 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:22.931+0000 7febbffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:23:22.910 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:22.931+0000 7febbffff640 10 monclient(hunting): my global_id is 4380 2024-03-29T11:23:22.911 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:22.931+0000 7febbffff640 1 -- 172.21.15.84:0/3660444962 --> v1:172.21.15.84:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7febb4003520 con 0x7febc80b38c0 2024-03-29T11:23:22.911 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:22.931+0000 7febbffff640 1 -- 172.21.15.84:0/3660444962 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3322762342 0 0) 0x7febc8142d30 con 0x7febc80a9c00 2024-03-29T11:23:22.911 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:22.931+0000 7febbffff640 10 monclient(hunting): _negotiate 2024-03-29T11:23:22.911 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:22.931+0000 7febbffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:23:22.911 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:22.931+0000 7febbffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:23:22.911 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:22.931+0000 7febbffff640 10 monclient(hunting): my global_id is 4376 2024-03-29T11:23:22.911 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:22.931+0000 7febbffff640 1 -- 172.21.15.84:0/3660444962 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7febc8141eb0 con 0x7febc80a9c00 2024-03-29T11:23:22.911 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:22.931+0000 7febbffff640 1 -- 172.21.15.84:0/3660444962 <== mon.0 v1:172.21.15.84:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 772251747 0 0) 0x7febb4003520 con 0x7febc80b38c0 2024-03-29T11:23:22.911 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:22.931+0000 7febbffff640 1 -- 172.21.15.84:0/3660444962 >> v1:172.21.15.84:6790/0 conn(0x7febc80a9c00 legacy=0x7febc813bb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:22.911 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:22.932+0000 7febbffff640 1 -- 172.21.15.84:0/3660444962 >> v1:172.21.15.179:6789/0 conn(0x7febc80b2800 legacy=0x7febc8142620 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:23:22.912 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:22.933+0000 7febbffff640 10 monclient: _finish_hunting 0 2024-03-29T11:23:22.912 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:22.933+0000 7febbffff640 1 monclient: found mon.noname-a 2024-03-29T11:23:22.912 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:22.933+0000 7febbffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:23:22.912 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:22.933+0000 7febbffff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.84:6789/0 2024-03-29T11:23:22.912 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:22.933+0000 7febbffff640 1 -- 172.21.15.84:0/3660444962 --> v1:172.21.15.84:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7febc80aaf90 con 0x7febc80b38c0 2024-03-29T11:23:22.912 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:22.933+0000 7febbffff640 10 monclient: _finish_auth 0 2024-03-29T11:23:22.912 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:22.933+0000 7febbffff640 10 monclient: _check_auth_tickets 2024-03-29T11:23:22.912 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:22.933+0000 7febbffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:23:22.913 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:22.933+0000 7febbffff640 1 -- 172.21.15.84:0/3660444962 <== mon.0 v1:172.21.15.84:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4236868694 0 0) 0x7febb8001690 con 0x7febc80b38c0 2024-03-29T11:23:22.913 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:22.933+0000 7febbffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:23:22.913 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:22.933+0000 7febbffff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:23:22.913 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:22.933+0000 7febbffff640 10 monclient: dump: 2024-03-29T11:23:22.913 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T11:23:22.913 INFO:teuthology.misc.health.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:23:22.913 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T11:18:53.864025+0000 2024-03-29T11:23:22.913 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T11:18:53.864025+0000 2024-03-29T11:23:22.913 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T11:23:22.913 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:23:22.913 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T11:23:22.913 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T11:23:22.913 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T11:23:22.913 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:23:22.913 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:22.933+0000 7febce458640 5 monclient: authenticate success, global_id 4380 2024-03-29T11:23:22.913 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:22.933+0000 7febce458640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:23:22.914 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:22.933+0000 7febbffff640 1 -- 172.21.15.84:0/3660444962 <== mon.0 v1:172.21.15.84:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7febb80017f0 con 0x7febc80b38c0 2024-03-29T11:23:22.914 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:22.933+0000 7febbffff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:23:22.914 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:22.933+0000 7febbffff640 1 -- 172.21.15.84:0/3660444962 <== mon.0 v1:172.21.15.84:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4236868694 0 0) 0x7febb8001bc0 con 0x7febc80b38c0 2024-03-29T11:23:22.914 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:22.933+0000 7febbffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:23:22.914 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:22.933+0000 7febbffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:23:22.914 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:22.933+0000 7febbffff640 10 monclient: dump: 2024-03-29T11:23:22.914 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T11:23:22.914 INFO:teuthology.misc.health.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:23:22.914 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T11:18:53.864025+0000 2024-03-29T11:23:22.914 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T11:18:53.864025+0000 2024-03-29T11:23:22.914 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T11:23:22.914 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:23:22.914 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T11:23:22.914 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T11:23:22.915 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T11:23:22.915 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:23:22.915 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:22.933+0000 7febce458640 10 monclient: get_monmap_and_config success 2024-03-29T11:23:22.915 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:22.933+0000 7febce458640 4 set_mon_vals no callback set 2024-03-29T11:23:22.916 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:22.934+0000 7febce458640 10 monclient: shutdown 2024-03-29T11:23:22.916 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:22.934+0000 7febce458640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:23:22.916 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:22.934+0000 7febce458640 1 -- 172.21.15.84:0/3660444962 >> v1:172.21.15.84:6789/0 conn(0x7febc80b38c0 legacy=0x7febc814d7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:22.916 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:22.936+0000 7febce458640 1 -- 172.21.15.84:0/3660444962 shutdown_connections 2024-03-29T11:23:22.916 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:22.936+0000 7febce458640 1 -- 172.21.15.84:0/3660444962 >> 172.21.15.84:0/3660444962 conn(0x7febc8066a00 msgr2=0x7febc8066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:23:22.920 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:22.938+0000 7febce458640 1 -- 172.21.15.84:0/3660444962 shutdown_connections 2024-03-29T11:23:22.920 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:22.940+0000 7febce458640 1 -- 172.21.15.84:0/3660444962 wait complete. 2024-03-29T11:23:22.920 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:22.941+0000 7febce458640 10 monclient: build_initial_monmap 2024-03-29T11:23:22.920 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:22.941+0000 7febce458640 10 monclient: monmap: 2024-03-29T11:23:22.920 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-29T11:23:22.920 INFO:teuthology.misc.health.smithi084.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:23:22.920 INFO:teuthology.misc.health.smithi084.stderr:last_changed 0.000000 2024-03-29T11:23:22.921 INFO:teuthology.misc.health.smithi084.stderr:created 0.000000 2024-03-29T11:23:22.921 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-29T11:23:22.921 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:23:22.921 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-29T11:23:22.921 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.noname-b 2024-03-29T11:23:22.921 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-c 2024-03-29T11:23:22.921 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:23:22.922 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:22.942+0000 7febce458640 1 librados: starting msgr at 2024-03-29T11:23:22.922 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:22.942+0000 7febce458640 1 librados: starting objecter 2024-03-29T11:23:22.922 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:22.942+0000 7febce458640 1 Processor -- start 2024-03-29T11:23:22.922 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:22.942+0000 7febce458640 1 -- start start 2024-03-29T11:23:22.923 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:22.942+0000 7febce458640 1 librados: setting wanted keys 2024-03-29T11:23:22.923 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:22.942+0000 7febce458640 1 librados: calling monclient init 2024-03-29T11:23:22.923 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:22.943+0000 7febce458640 10 monclient: init 2024-03-29T11:23:22.923 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:22.943+0000 7febce458640 10 monclient: _reopen_session rank -1 2024-03-29T11:23:22.923 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:22.944+0000 7febce458640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:23:22.923 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:22.944+0000 7febce458640 10 monclient(hunting): picked mon.noname-b con 0x7febc80a9c00 addr v1:172.21.15.179:6789/0 2024-03-29T11:23:22.926 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:22.947+0000 7febce458640 10 monclient(hunting): picked mon.noname-a con 0x7febc80b2800 addr v1:172.21.15.84:6789/0 2024-03-29T11:23:22.930 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:22.947+0000 7febce458640 10 monclient(hunting): picked mon.noname-c con 0x7febc80b38c0 addr v1:172.21.15.84:6790/0 2024-03-29T11:23:22.930 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:22.948+0000 7febccc55640 1 --1- >> v1:172.21.15.84:6789/0 conn(0x7febc80b2800 0x7febc80ad990 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.84:6789/0 says I am v1:172.21.15.84:40865/0 (socket says 172.21.15.84:40865) 2024-03-29T11:23:22.930 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:22.950+0000 7febccc55640 1 -- 172.21.15.84:0/1086049121 learned_addr learned my addr 172.21.15.84:0/1086049121 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-29T11:23:22.930 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:22.950+0000 7febce458640 1 -- 172.21.15.84:0/1086049121 --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7febc81d79c0 con 0x7febc80b2800 2024-03-29T11:23:22.930 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:22.950+0000 7febce458640 1 -- 172.21.15.84:0/1086049121 --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7febc80ace90 con 0x7febc80a9c00 2024-03-29T11:23:22.930 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:22.950+0000 7febce458640 1 -- 172.21.15.84:0/1086049121 --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7febc80ad020 con 0x7febc80b38c0 2024-03-29T11:23:22.930 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:22.950+0000 7febce458640 10 monclient(hunting): _renew_subs 2024-03-29T11:23:22.930 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:22.950+0000 7febce458640 10 monclient(hunting): authenticate will time out at 1064.244963s 2024-03-29T11:23:22.930 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:22.950+0000 7febcd456640 1 --1- >> v1:172.21.15.179:6789/0 conn(0x7febc80a9c00 0x7febc81d71d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.179:6789/0 says I am v1:172.21.15.84:39671/0 (socket says 172.21.15.84:39671) 2024-03-29T11:23:22.930 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:22.950+0000 7febbdffb640 10 client.?.objecter ms_handle_connect 0x7febc80b38c0 2024-03-29T11:23:22.930 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:22.951+0000 7febbdffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:23:22.931 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:22.951+0000 7febbdffb640 10 client.?.objecter ms_handle_connect 0x7febc80b2800 2024-03-29T11:23:22.931 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:22.951+0000 7febbdffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:23:22.931 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:22.951+0000 7febbdffb640 1 -- 172.21.15.84:0/1086049121 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2551740638 0 0) 0x7febc80ad020 con 0x7febc80b38c0 2024-03-29T11:23:22.931 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:22.951+0000 7febbdffb640 10 monclient(hunting): _negotiate 2024-03-29T11:23:22.931 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:22.951+0000 7febbdffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:23:22.931 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:22.951+0000 7febbdffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:23:22.931 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:22.951+0000 7febbdffb640 10 monclient(hunting): my global_id is 4379 2024-03-29T11:23:22.931 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:22.951+0000 7febbdffb640 1 -- 172.21.15.84:0/1086049121 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7febac004eb0 con 0x7febc80b38c0 2024-03-29T11:23:22.931 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:22.951+0000 7febbdffb640 1 -- 172.21.15.84:0/1086049121 <== mon.0 v1:172.21.15.84:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2066700830 0 0) 0x7febc81d79c0 con 0x7febc80b2800 2024-03-29T11:23:22.931 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:22.951+0000 7febbdffb640 10 monclient(hunting): _negotiate 2024-03-29T11:23:22.931 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:22.951+0000 7febbdffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:23:22.931 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:22.951+0000 7febbdffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:23:22.931 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:22.951+0000 7febbdffb640 10 monclient(hunting): my global_id is 4383 2024-03-29T11:23:22.932 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:22.951+0000 7febbdffb640 1 -- 172.21.15.84:0/1086049121 --> v1:172.21.15.84:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7febac0051d0 con 0x7febc80b2800 2024-03-29T11:23:22.932 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:22.951+0000 7febbdffb640 1 -- 172.21.15.84:0/1086049121 <== mon.0 v1:172.21.15.84:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 696073534 0 0) 0x7febac0051d0 con 0x7febc80b2800 2024-03-29T11:23:22.932 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:22.951+0000 7febbdffb640 1 -- 172.21.15.84:0/1086049121 --> v1:172.21.15.84:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7febc81d79c0 con 0x7febc80b2800 2024-03-29T11:23:22.932 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:22.951+0000 7febbdffb640 1 -- 172.21.15.84:0/1086049121 <== mon.0 v1:172.21.15.84:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4236868694 0 0) 0x7febc40019f0 con 0x7febc80b2800 2024-03-29T11:23:22.932 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:22.951+0000 7febbdffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:23:22.933 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:22.951+0000 7febbdffb640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:23:22.933 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:22.951+0000 7febbdffb640 10 monclient(hunting): dump: 2024-03-29T11:23:22.933 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T11:23:22.933 INFO:teuthology.misc.health.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:23:22.933 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T11:18:53.864025+0000 2024-03-29T11:23:22.933 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T11:18:53.864025+0000 2024-03-29T11:23:22.933 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T11:23:22.933 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:23:22.933 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T11:23:22.933 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T11:23:22.933 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T11:23:22.933 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:23:22.933 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:22.951+0000 7febbdffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:23:22.933 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:22.951+0000 7febbdffb640 10 client.?.objecter ms_handle_connect 0x7febc80a9c00 2024-03-29T11:23:22.934 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:22.951+0000 7febbdffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:23:22.934 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:22.951+0000 7febbdffb640 1 -- 172.21.15.84:0/1086049121 <== mon.2 v1:172.21.15.84:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3303834459 0 0) 0x7febac004eb0 con 0x7febc80b38c0 2024-03-29T11:23:22.934 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:22.952+0000 7febbdffb640 1 -- 172.21.15.84:0/1086049121 --> v1:172.21.15.84:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7febac0051d0 con 0x7febc80b38c0 2024-03-29T11:23:22.934 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:22.952+0000 7febbdffb640 1 -- 172.21.15.84:0/1086049121 <== mon.2 v1:172.21.15.84:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4236868694 0 0) 0x7febb8001b00 con 0x7febc80b38c0 2024-03-29T11:23:22.934 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:22.952+0000 7febbdffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:23:22.934 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:22.952+0000 7febbdffb640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:23:22.934 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:22.952+0000 7febbdffb640 10 monclient(hunting): dump: 2024-03-29T11:23:22.934 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T11:23:22.934 INFO:teuthology.misc.health.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:23:22.934 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T11:18:53.864025+0000 2024-03-29T11:23:22.934 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T11:18:53.864025+0000 2024-03-29T11:23:22.934 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T11:23:22.934 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:23:22.934 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T11:23:22.934 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T11:23:22.935 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T11:23:22.935 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:23:22.935 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:22.952+0000 7febbdffb640 1 -- 172.21.15.84:0/1086049121 <== mon.1 v1:172.21.15.179:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1399991379 0 0) 0x7febc80ace90 con 0x7febc80a9c00 2024-03-29T11:23:22.935 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:22.952+0000 7febbdffb640 10 monclient(hunting): _negotiate 2024-03-29T11:23:22.935 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:22.952+0000 7febbdffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:23:22.935 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:22.952+0000 7febbdffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:23:22.935 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:22.952+0000 7febbdffb640 10 monclient(hunting): my global_id is 4342 2024-03-29T11:23:22.935 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:22.952+0000 7febbdffb640 1 -- 172.21.15.84:0/1086049121 --> v1:172.21.15.179:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7febac005da0 con 0x7febc80a9c00 2024-03-29T11:23:22.935 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:22.952+0000 7febbdffb640 1 -- 172.21.15.84:0/1086049121 <== mon.0 v1:172.21.15.84:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1418569993 0 0) 0x7febc81d79c0 con 0x7febc80b2800 2024-03-29T11:23:22.935 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:22.952+0000 7febbdffb640 1 -- 172.21.15.84:0/1086049121 >> v1:172.21.15.84:6790/0 conn(0x7febc80b38c0 legacy=0x7febc81e1ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:22.935 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:22.952+0000 7febbdffb640 1 -- 172.21.15.84:0/1086049121 >> v1:172.21.15.179:6789/0 conn(0x7febc80a9c00 legacy=0x7febc81d71d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:22.935 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:22.952+0000 7febbdffb640 10 monclient: _finish_hunting 0 2024-03-29T11:23:22.935 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:22.952+0000 7febbdffb640 1 monclient: found mon.a 2024-03-29T11:23:22.935 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:22.952+0000 7febbdffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:23:22.935 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:22.952+0000 7febbdffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.84:6789/0 2024-03-29T11:23:22.935 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:22.952+0000 7febbdffb640 1 -- 172.21.15.84:0/1086049121 --> v1:172.21.15.84:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7febc81d7ee0 con 0x7febc80b2800 2024-03-29T11:23:22.936 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:22.952+0000 7febbdffb640 10 monclient: _finish_auth 0 2024-03-29T11:23:22.936 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:22.952+0000 7febbdffb640 10 monclient: _check_auth_tickets 2024-03-29T11:23:22.936 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:22.952+0000 7febbdffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:23:22.936 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:22.952+0000 7febce458640 5 monclient: authenticate success, global_id 4383 2024-03-29T11:23:22.936 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:22.952+0000 7febce458640 10 monclient: _renew_subs 2024-03-29T11:23:22.936 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:22.952+0000 7febce458640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.84:6789/0 2024-03-29T11:23:22.936 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:22.952+0000 7febce458640 1 -- 172.21.15.84:0/1086049121 --> v1:172.21.15.84:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7febc80b07f0 con 0x7febc80b2800 2024-03-29T11:23:22.936 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:22.952+0000 7febce458640 10 client.4383.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:23:22.936 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:22.952+0000 7febce458640 10 monclient: _renew_subs 2024-03-29T11:23:22.936 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:22.952+0000 7febce458640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.84:6789/0 2024-03-29T11:23:22.936 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:22.952+0000 7febce458640 1 -- 172.21.15.84:0/1086049121 --> v1:172.21.15.84:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7febc81e3200 con 0x7febc80b2800 2024-03-29T11:23:22.936 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:22.952+0000 7febce458640 1 librados: init done 2024-03-29T11:23:22.938 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:22.954+0000 7febbdffb640 1 -- 172.21.15.84:0/1086049121 <== mon.0 v1:172.21.15.84:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7febc4003610 con 0x7febc80b2800 2024-03-29T11:23:22.938 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:22.954+0000 7febbdffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:23:22.938 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:22.954+0000 7febbdffb640 1 -- 172.21.15.84:0/1086049121 <== mon.0 v1:172.21.15.84:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4236868694 0 0) 0x7febc4003960 con 0x7febc80b2800 2024-03-29T11:23:22.938 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:22.954+0000 7febbdffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:23:22.939 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:22.954+0000 7febbdffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:23:22.939 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:22.954+0000 7febbdffb640 10 monclient: dump: 2024-03-29T11:23:22.939 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T11:23:22.939 INFO:teuthology.misc.health.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:23:22.939 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T11:18:53.864025+0000 2024-03-29T11:23:22.939 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T11:18:53.864025+0000 2024-03-29T11:23:22.939 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T11:23:22.939 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:23:22.939 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T11:23:22.939 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T11:23:22.939 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T11:23:22.939 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:23:22.939 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:22.954+0000 7febbeffd640 4 set_mon_vals no callback set 2024-03-29T11:23:22.939 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:22.955+0000 7febbdffb640 1 -- 172.21.15.84:0/1086049121 <== mon.0 v1:172.21.15.84:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4033942909 0 0) 0x7febc4003b80 con 0x7febc80b2800 2024-03-29T11:23:22.939 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:22.956+0000 7febbdffb640 1 -- 172.21.15.84:0/1086049121 <== mon.0 v1:172.21.15.84:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2808139587 0 0) 0x7febc4084a30 con 0x7febc80b2800 2024-03-29T11:23:22.940 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:22.956+0000 7febbdffb640 10 client.4383.objecter ms_dispatch 0x7febc8000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:23:22.940 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:22.956+0000 7febbdffb640 3 client.4383.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:23:22.940 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:22.956+0000 7febbdffb640 3 client.4383.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:23:22.940 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:22.956+0000 7febbdffb640 20 client.4383.objecter dump_active .. 0 homeless 2024-03-29T11:23:22.940 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:22.956+0000 7febb37fe640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:23:22.940 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:22.956+0000 7febb37fe640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:23:22.940 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:22.956+0000 7febb37fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.84:6789/0 2024-03-29T11:23:22.940 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:22.956+0000 7febb37fe640 1 -- 172.21.15.84:0/1086049121 --> v1:172.21.15.84:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feba8005080 con 0x7febc80b2800 2024-03-29T11:23:22.940 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:22.957+0000 7febcd456640 10 monclient: get_auth_request con 0x7febac06e470 auth_method 0 2024-03-29T11:23:22.940 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:22.958+0000 7febbdffb640 10 client.4383.objecter ms_handle_connect 0x7febac06e470 2024-03-29T11:23:22.943 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:22.964+0000 7febbdffb640 1 -- 172.21.15.84:0/1086049121 <== mon.0 v1:172.21.15.84:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7febc4050310 con 0x7febc80b2800 2024-03-29T11:23:22.943 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:22.964+0000 7febbdffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:23:22.943 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:22.964+0000 7febbdffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:23:23.273 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:23.294+0000 7febb37fe640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:23:23.273 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:23.294+0000 7febb37fe640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:23:23.273 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:23.294+0000 7febb37fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.84:6789/0 2024-03-29T11:23:23.273 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:23.294+0000 7febb37fe640 1 -- 172.21.15.84:0/1086049121 --> v1:172.21.15.84:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7feba8005330 con 0x7febc80b2800 2024-03-29T11:23:23.274 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:23.296+0000 7febbdffb640 1 -- 172.21.15.84:0/1086049121 <== mon.0 v1:172.21.15.84:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4209477549) 0x7febc4052240 con 0x7febc80b2800 2024-03-29T11:23:23.274 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:23.296+0000 7febbdffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:23:23.274 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:23.296+0000 7febbdffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:23:23.276 INFO:teuthology.misc.health.smithi084.stdout: 2024-03-29T11:23:23.276 INFO:teuthology.misc.health.smithi084.stdout:{"epoch":10,"fsid":"2c3e002f-9eb0-4860-853b-4ef1ef562213","created":"2024-03-29T11:19:32.029169+0000","modified":"2024-03-29T11:19:34.200299+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:34.200299+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"5c39e276-2034-4716-bbc0-57a634eefa92","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"cd580e99-109f-4bf1-88c0-a5987bc8b9ed","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"11e60a5a-051d-453f-a190-66387cb35592","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"2c329fbe-bbc4-4030-bf87-d711cad04679","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f4213d4a-bd36-4433-91ae-d7e2b02481d3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"f0cf44ea-6055-48af-805b-aed858102ff9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"d895eaa0-cc0b-4ee4-9b5a-2dfcd89a60b7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"9bc5a7bc-8f52-4db9-96d6-de5e68d52eaa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:23:23.278 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:23.300+0000 7febb37fe640 10 librados: watch_flush enter 2024-03-29T11:23:23.279 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:23.300+0000 7febb37fe640 10 librados: watch_flush exit 2024-03-29T11:23:23.279 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:23.300+0000 7febb37fe640 20 client.4383.objecter shutdown clearing up homeless session... 2024-03-29T11:23:23.279 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:23.300+0000 7febb37fe640 10 client.4383.objecter successfully canceled tick 2024-03-29T11:23:23.279 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:23.300+0000 7febb37fe640 1 -- 172.21.15.84:0/1086049121 >> v1:172.21.15.179:6816/2981134980 conn(0x7febac06e470 legacy=0x7febac070930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:23.279 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:23.300+0000 7febb37fe640 10 monclient: shutdown 2024-03-29T11:23:23.279 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:23.300+0000 7febb37fe640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:23:23.279 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:23.300+0000 7febb37fe640 1 -- 172.21.15.84:0/1086049121 >> v1:172.21.15.84:6789/0 conn(0x7febc80b2800 legacy=0x7febc80ad990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:23.280 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:23.302+0000 7febb37fe640 1 -- 172.21.15.84:0/1086049121 shutdown_connections 2024-03-29T11:23:23.280 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:23.302+0000 7febb37fe640 1 -- 172.21.15.84:0/1086049121 >> 172.21.15.84:0/1086049121 conn(0x7febc8066a00 msgr2=0x7febc8145240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:23:23.281 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:23.302+0000 7febb37fe640 1 -- 172.21.15.84:0/1086049121 shutdown_connections 2024-03-29T11:23:23.281 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:23.302+0000 7febb37fe640 1 -- 172.21.15.84:0/1086049121 wait complete. 2024-03-29T11:23:23.281 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:23.302+0000 7febb37fe640 1 librados: shutdown 2024-03-29T11:23:23.300 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:23:30.107 DEBUG:teuthology.orchestra.run.smithi084:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:23:30.264 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:30.285+0000 7f6a05c4e640 10 monclient: get_monmap_and_config 2024-03-29T11:23:30.264 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:30.285+0000 7f6a05c4e640 10 monclient: build_initial_monmap 2024-03-29T11:23:30.264 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:30.285+0000 7f6a05c4e640 10 monclient: monmap: 2024-03-29T11:23:30.264 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-29T11:23:30.264 INFO:teuthology.misc.health.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:23:30.264 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T11:23:30.286067+0000 2024-03-29T11:23:30.265 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T11:23:30.286067+0000 2024-03-29T11:23:30.265 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-29T11:23:30.265 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:23:30.265 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-29T11:23:30.265 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.noname-c 2024-03-29T11:23:30.265 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-b 2024-03-29T11:23:30.265 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:23:30.268 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:30.287+0000 7f6a05c4e640 1 Processor -- start 2024-03-29T11:23:30.268 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:30.289+0000 7f6a05c4e640 1 -- start start 2024-03-29T11:23:30.269 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:30.289+0000 7f6a05c4e640 10 monclient: init 2024-03-29T11:23:30.271 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:30.290+0000 7f6a05c4e640 10 monclient: _reopen_session rank -1 2024-03-29T11:23:30.271 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:30.290+0000 7f6a05c4e640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:23:30.272 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:30.290+0000 7f6a05c4e640 10 monclient(hunting): picked mon.noname-a con 0x7f6a00071de0 addr v1:172.21.15.84:6789/0 2024-03-29T11:23:30.272 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:30.290+0000 7f6a05c4e640 10 monclient(hunting): picked mon.noname-b con 0x7f6a000721c0 addr v1:172.21.15.84:6790/0 2024-03-29T11:23:30.272 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:30.290+0000 7f6a05c4e640 10 monclient(hunting): picked mon.noname-c con 0x7f6a00145ae0 addr v1:172.21.15.179:6789/0 2024-03-29T11:23:30.272 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:30.290+0000 7f6a05c4e640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6a00145650 con 0x7f6a00071de0 2024-03-29T11:23:30.272 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:30.290+0000 7f6a05c4e640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6a000ae260 con 0x7f6a00145ae0 2024-03-29T11:23:30.272 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:30.290+0000 7f6a05c4e640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6a001420d0 con 0x7f6a000721c0 2024-03-29T11:23:30.272 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:30.290+0000 7f6a05c4e640 10 monclient(hunting): _renew_subs 2024-03-29T11:23:30.272 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:30.290+0000 7f6a05c4e640 10 monclient(hunting): authenticate will time out at 1071.584950s 2024-03-29T11:23:30.274 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:30.294+0000 7f69ff7fe640 1 --1- >> v1:172.21.15.84:6789/0 conn(0x7f6a00071de0 0x7f6a000aed80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.84:6789/0 says I am v1:172.21.15.84:48307/0 (socket says 172.21.15.84:48307) 2024-03-29T11:23:30.274 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:30.294+0000 7f69ff7fe640 1 -- 172.21.15.84:0/4248291848 learned_addr learned my addr 172.21.15.84:0/4248291848 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-29T11:23:30.274 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:30.294+0000 7f69fe7fc640 1 -- 172.21.15.84:0/4248291848 <== mon.0 v1:172.21.15.84:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2317493415 0 0) 0x7f6a00145650 con 0x7f6a00071de0 2024-03-29T11:23:30.274 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:30.294+0000 7f69fe7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:23:30.274 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:30.294+0000 7f69fe7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:23:30.274 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:30.294+0000 7f69fe7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:23:30.274 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:30.294+0000 7f69fe7fc640 10 monclient(hunting): my global_id is 4386 2024-03-29T11:23:30.275 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:30.294+0000 7f69fe7fc640 1 -- 172.21.15.84:0/4248291848 --> v1:172.21.15.84:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f69ec003520 con 0x7f6a00071de0 2024-03-29T11:23:30.275 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:30.294+0000 7f69fe7fc640 1 -- 172.21.15.84:0/4248291848 <== mon.0 v1:172.21.15.84:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2313151905 0 0) 0x7f69ec003520 con 0x7f6a00071de0 2024-03-29T11:23:30.275 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:30.294+0000 7f69fe7fc640 1 -- 172.21.15.84:0/4248291848 >> v1:172.21.15.84:6790/0 conn(0x7f6a000721c0 legacy=0x7f6a0013f400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:23:30.275 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:30.294+0000 7f69fe7fc640 1 -- 172.21.15.84:0/4248291848 >> v1:172.21.15.179:6789/0 conn(0x7f6a00145ae0 legacy=0x7f6a00145f50 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-29T11:23:30.275 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:30.294+0000 7f69fe7fc640 10 monclient: _finish_hunting 0 2024-03-29T11:23:30.275 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:30.294+0000 7f69fe7fc640 1 monclient: found mon.noname-a 2024-03-29T11:23:30.275 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:30.294+0000 7f69fe7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:23:30.275 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:30.294+0000 7f69fe7fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.84:6789/0 2024-03-29T11:23:30.275 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:30.294+0000 7f69fe7fc640 1 -- 172.21.15.84:0/4248291848 --> v1:172.21.15.84:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a000b0590 con 0x7f6a00071de0 2024-03-29T11:23:30.275 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:30.294+0000 7f69fe7fc640 10 monclient: _finish_auth 0 2024-03-29T11:23:30.275 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:30.294+0000 7f69fe7fc640 10 monclient: _check_auth_tickets 2024-03-29T11:23:30.275 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:30.294+0000 7f69fe7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:23:30.275 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:30.295+0000 7f69fe7fc640 1 -- 172.21.15.84:0/4248291848 <== mon.0 v1:172.21.15.84:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4236868694 0 0) 0x7f69f8001670 con 0x7f6a00071de0 2024-03-29T11:23:30.276 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:30.295+0000 7f69fe7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:23:30.276 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:30.295+0000 7f69fe7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:23:30.276 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:30.295+0000 7f69fe7fc640 10 monclient: dump: 2024-03-29T11:23:30.276 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T11:23:30.276 INFO:teuthology.misc.health.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:23:30.276 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T11:18:53.864025+0000 2024-03-29T11:23:30.276 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T11:18:53.864025+0000 2024-03-29T11:23:30.276 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T11:23:30.276 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:23:30.276 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T11:23:30.276 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T11:23:30.276 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T11:23:30.277 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:23:30.277 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:30.295+0000 7f6a05c4e640 5 monclient: authenticate success, global_id 4386 2024-03-29T11:23:30.277 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:30.295+0000 7f6a05c4e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:23:30.277 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:30.295+0000 7f69fe7fc640 1 -- 172.21.15.84:0/4248291848 <== mon.0 v1:172.21.15.84:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f69f80018d0 con 0x7f6a00071de0 2024-03-29T11:23:30.277 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:30.295+0000 7f69fe7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:23:30.277 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:30.295+0000 7f6a05c4e640 10 monclient: get_monmap_and_config success 2024-03-29T11:23:30.277 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:30.295+0000 7f6a05c4e640 4 set_mon_vals no callback set 2024-03-29T11:23:30.278 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:30.296+0000 7f6a05c4e640 10 monclient: shutdown 2024-03-29T11:23:30.278 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:30.296+0000 7f6a05c4e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:23:30.278 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:30.296+0000 7f6a05c4e640 1 -- 172.21.15.84:0/4248291848 >> v1:172.21.15.84:6789/0 conn(0x7f6a00071de0 legacy=0x7f6a000aed80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:30.278 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:30.296+0000 7f69fe7fc640 1 -- 172.21.15.84:0/4248291848 <== mon.0 v1:172.21.15.84:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4236868694 0 0) 0x7f69f8001670 con 0x7f6a00071de0 2024-03-29T11:23:30.278 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:30.296+0000 7f69fe7fc640 10 monclient: discarding stray monitor message mon_map magic: 0 v1 2024-03-29T11:23:30.278 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:30.296+0000 7f6a05c4e640 1 -- 172.21.15.84:0/4248291848 shutdown_connections 2024-03-29T11:23:30.278 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:30.296+0000 7f6a05c4e640 1 -- 172.21.15.84:0/4248291848 >> 172.21.15.84:0/4248291848 conn(0x7f6a0006b660 msgr2=0x7f6a0006ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:23:30.279 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:30.297+0000 7f6a05c4e640 1 -- 172.21.15.84:0/4248291848 shutdown_connections 2024-03-29T11:23:30.280 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:30.297+0000 7f6a05c4e640 1 -- 172.21.15.84:0/4248291848 wait complete. 2024-03-29T11:23:30.280 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:30.298+0000 7f6a05c4e640 10 monclient: build_initial_monmap 2024-03-29T11:23:30.280 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:30.298+0000 7f6a05c4e640 10 monclient: monmap: 2024-03-29T11:23:30.280 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-29T11:23:30.280 INFO:teuthology.misc.health.smithi084.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:23:30.280 INFO:teuthology.misc.health.smithi084.stderr:last_changed 0.000000 2024-03-29T11:23:30.280 INFO:teuthology.misc.health.smithi084.stderr:created 0.000000 2024-03-29T11:23:30.280 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-29T11:23:30.280 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:23:30.281 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-29T11:23:30.281 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.noname-b 2024-03-29T11:23:30.281 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-c 2024-03-29T11:23:30.281 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:23:30.281 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:30.299+0000 7f6a05c4e640 1 librados: starting msgr at 2024-03-29T11:23:30.281 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:30.299+0000 7f6a05c4e640 1 librados: starting objecter 2024-03-29T11:23:30.281 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:30.299+0000 7f6a05c4e640 1 Processor -- start 2024-03-29T11:23:30.281 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:30.299+0000 7f6a05c4e640 1 -- start start 2024-03-29T11:23:30.281 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:30.299+0000 7f6a05c4e640 1 librados: setting wanted keys 2024-03-29T11:23:30.281 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:30.299+0000 7f6a05c4e640 1 librados: calling monclient init 2024-03-29T11:23:30.281 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:30.299+0000 7f6a05c4e640 10 monclient: init 2024-03-29T11:23:30.281 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:30.300+0000 7f6a05c4e640 10 monclient: _reopen_session rank -1 2024-03-29T11:23:30.281 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:30.300+0000 7f6a05c4e640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:23:30.281 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:30.300+0000 7f6a05c4e640 10 monclient(hunting): picked mon.noname-c con 0x7f6a00071de0 addr v1:172.21.15.84:6790/0 2024-03-29T11:23:30.281 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:30.300+0000 7f6a05c4e640 10 monclient(hunting): picked mon.noname-a con 0x7f6a00145ae0 addr v1:172.21.15.84:6789/0 2024-03-29T11:23:30.282 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:30.300+0000 7f6a05c4e640 10 monclient(hunting): picked mon.noname-b con 0x7f6a00132560 addr v1:172.21.15.179:6789/0 2024-03-29T11:23:30.282 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:30.300+0000 7f6a05c4e640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6a0006f840 con 0x7f6a00145ae0 2024-03-29T11:23:30.282 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:30.300+0000 7f6a05c4e640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6a00071800 con 0x7f6a00132560 2024-03-29T11:23:30.282 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:30.300+0000 7f6a05c4e640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6a00071520 con 0x7f6a00071de0 2024-03-29T11:23:30.282 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:30.300+0000 7f6a05c4e640 10 monclient(hunting): _renew_subs 2024-03-29T11:23:30.282 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:30.300+0000 7f6a05c4e640 10 monclient(hunting): authenticate will time out at 1071.594721s 2024-03-29T11:23:30.287 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:30.308+0000 7f69ff7fe640 1 --1- >> v1:172.21.15.84:6790/0 conn(0x7f6a00071de0 0x7f6a00070e10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.84:6790/0 says I am v1:172.21.15.84:51231/0 (socket says 172.21.15.84:51231) 2024-03-29T11:23:30.287 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:30.308+0000 7f69fffff640 1 --1- >> v1:172.21.15.179:6789/0 conn(0x7f6a00132560 0x7f6a001351f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.179:6789/0 says I am v1:172.21.15.84:60181/0 (socket says 172.21.15.84:60181) 2024-03-29T11:23:30.287 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:30.308+0000 7f69fffff640 1 -- 172.21.15.84:0/2834156928 learned_addr learned my addr 172.21.15.84:0/2834156928 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-29T11:23:30.287 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:30.308+0000 7f69f7fff640 10 client.?.objecter ms_handle_connect 0x7f6a00132560 2024-03-29T11:23:30.287 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:30.308+0000 7f69f7fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:23:30.288 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:30.308+0000 7f69f7fff640 1 -- 172.21.15.84:0/2834156928 <== mon.1 v1:172.21.15.179:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 200638613 0 0) 0x7f6a00071800 con 0x7f6a00132560 2024-03-29T11:23:30.288 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:30.308+0000 7f69f7fff640 10 monclient(hunting): _negotiate 2024-03-29T11:23:30.288 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:30.308+0000 7f69f7fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:23:30.288 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:30.308+0000 7f69f7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:23:30.288 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:30.308+0000 7f69f7fff640 10 monclient(hunting): my global_id is 4345 2024-03-29T11:23:30.288 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:30.308+0000 7f69f7fff640 1 -- 172.21.15.84:0/2834156928 --> v1:172.21.15.179:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f69e0004eb0 con 0x7f6a00132560 2024-03-29T11:23:30.288 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:30.309+0000 7f69f7fff640 1 -- 172.21.15.84:0/2834156928 <== mon.1 v1:172.21.15.179:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3889390307 0 0) 0x7f69e0004eb0 con 0x7f6a00132560 2024-03-29T11:23:30.288 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:30.309+0000 7f69f7fff640 1 -- 172.21.15.84:0/2834156928 --> v1:172.21.15.179:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6a00071800 con 0x7f6a00132560 2024-03-29T11:23:30.289 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:30.309+0000 7f69f7fff640 1 -- 172.21.15.84:0/2834156928 <== mon.1 v1:172.21.15.179:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4236868694 0 0) 0x7f69f0001b70 con 0x7f6a00132560 2024-03-29T11:23:30.289 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:30.309+0000 7f69f7fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:23:30.289 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:30.309+0000 7f69f7fff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:23:30.289 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:30.309+0000 7f69f7fff640 10 monclient(hunting): dump: 2024-03-29T11:23:30.289 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T11:23:30.289 INFO:teuthology.misc.health.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:23:30.289 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T11:18:53.864025+0000 2024-03-29T11:23:30.290 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T11:18:53.864025+0000 2024-03-29T11:23:30.290 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T11:23:30.290 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:23:30.290 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T11:23:30.290 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T11:23:30.290 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T11:23:30.290 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:23:30.290 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:30.309+0000 7f69f7fff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:23:30.291 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:30.311+0000 7f69f7fff640 10 client.?.objecter ms_handle_connect 0x7f6a00071de0 2024-03-29T11:23:30.291 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:30.311+0000 7f69f7fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:23:30.291 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:30.312+0000 7f69f7fff640 1 -- 172.21.15.84:0/2834156928 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2123055486 0 0) 0x7f6a00071520 con 0x7f6a00071de0 2024-03-29T11:23:30.291 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:30.312+0000 7f69f7fff640 10 monclient(hunting): _negotiate 2024-03-29T11:23:30.291 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:30.312+0000 7f69f7fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:23:30.291 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:30.312+0000 7f69f7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:23:30.291 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:30.312+0000 7f69f7fff640 10 monclient(hunting): my global_id is 4382 2024-03-29T11:23:30.292 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:30.312+0000 7f69f7fff640 1 -- 172.21.15.84:0/2834156928 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f69e0004eb0 con 0x7f6a00071de0 2024-03-29T11:23:30.292 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:30.312+0000 7f69f7fff640 1 -- 172.21.15.84:0/2834156928 <== mon.1 v1:172.21.15.179:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 89848234 0 0) 0x7f6a00071800 con 0x7f6a00132560 2024-03-29T11:23:30.292 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:30.312+0000 7f69f7fff640 1 -- 172.21.15.84:0/2834156928 >> v1:172.21.15.84:6790/0 conn(0x7f6a00071de0 legacy=0x7f6a00070e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:30.292 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:30.312+0000 7f69f7fff640 1 -- 172.21.15.84:0/2834156928 >> v1:172.21.15.84:6789/0 conn(0x7f6a00145ae0 legacy=0x7f6a00141290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:30.292 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:30.312+0000 7f69f7fff640 10 monclient: _finish_hunting 0 2024-03-29T11:23:30.292 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:30.312+0000 7f69f7fff640 1 monclient: found mon.b 2024-03-29T11:23:30.292 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:30.312+0000 7f69f7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:23:30.292 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:30.312+0000 7f69f7fff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-29T11:23:30.292 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:30.312+0000 7f69f7fff640 1 -- 172.21.15.84:0/2834156928 --> v1:172.21.15.179:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a001419a0 con 0x7f6a00132560 2024-03-29T11:23:30.292 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:30.312+0000 7f69f7fff640 10 monclient: _finish_auth 0 2024-03-29T11:23:30.292 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:30.312+0000 7f69f7fff640 10 monclient: _check_auth_tickets 2024-03-29T11:23:30.292 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:30.312+0000 7f69f7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:23:30.293 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:30.313+0000 7f6a05c4e640 5 monclient: authenticate success, global_id 4345 2024-03-29T11:23:30.293 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:30.313+0000 7f6a05c4e640 10 monclient: _renew_subs 2024-03-29T11:23:30.294 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:30.313+0000 7f6a05c4e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-29T11:23:30.294 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:30.313+0000 7f6a05c4e640 1 -- 172.21.15.84:0/2834156928 --> v1:172.21.15.179:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6a00135900 con 0x7f6a00132560 2024-03-29T11:23:30.294 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:30.313+0000 7f6a05c4e640 10 client.4345.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:23:30.294 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:30.313+0000 7f6a05c4e640 10 monclient: _renew_subs 2024-03-29T11:23:30.294 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:30.313+0000 7f6a05c4e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-29T11:23:30.294 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:30.313+0000 7f6a05c4e640 1 -- 172.21.15.84:0/2834156928 --> v1:172.21.15.179:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f6a00132990 con 0x7f6a00132560 2024-03-29T11:23:30.294 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:30.313+0000 7f6a05c4e640 1 librados: init done 2024-03-29T11:23:30.295 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:30.315+0000 7f69f7fff640 1 -- 172.21.15.84:0/2834156928 <== mon.1 v1:172.21.15.179:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f69f0002140 con 0x7f6a00132560 2024-03-29T11:23:30.296 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:30.315+0000 7f6a05c4e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:23:30.296 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:30.315+0000 7f69f7fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:23:30.296 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:30.315+0000 7f69f7fff640 1 -- 172.21.15.84:0/2834156928 <== mon.1 v1:172.21.15.179:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4236868694 0 0) 0x7f69f0004e20 con 0x7f6a00132560 2024-03-29T11:23:30.296 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:30.315+0000 7f69f7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:23:30.296 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:30.315+0000 7f69f7fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:23:30.296 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:30.315+0000 7f69f7fff640 10 monclient: dump: 2024-03-29T11:23:30.296 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T11:23:30.296 INFO:teuthology.misc.health.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:23:30.296 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T11:18:53.864025+0000 2024-03-29T11:23:30.296 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T11:18:53.864025+0000 2024-03-29T11:23:30.296 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T11:23:30.296 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:23:30.296 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T11:23:30.296 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T11:23:30.297 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T11:23:30.297 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:23:30.297 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:30.315+0000 7f69fd7fa640 4 set_mon_vals no callback set 2024-03-29T11:23:30.297 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:30.316+0000 7f6a05c4e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:23:30.297 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:30.316+0000 7f6a05c4e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-29T11:23:30.297 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:30.316+0000 7f6a05c4e640 1 -- 172.21.15.84:0/2834156928 --> v1:172.21.15.179:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6a00075f10 con 0x7f6a00132560 2024-03-29T11:23:30.297 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:30.316+0000 7f69f7fff640 1 -- 172.21.15.84:0/2834156928 <== mon.1 v1:172.21.15.179:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4033942909 0 0) 0x7f69f0004f80 con 0x7f6a00132560 2024-03-29T11:23:30.297 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:30.317+0000 7f69f7fff640 1 -- 172.21.15.84:0/2834156928 <== mon.1 v1:172.21.15.179:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2808139587 0 0) 0x7f69f0084e90 con 0x7f6a00132560 2024-03-29T11:23:30.297 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:30.317+0000 7f69f7fff640 10 client.4345.objecter ms_dispatch 0x7f6a00000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:23:30.299 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:30.320+0000 7f69f7fff640 3 client.4345.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:23:30.299 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:30.320+0000 7f69f7fff640 3 client.4345.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:23:30.299 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:30.320+0000 7f69f7fff640 20 client.4345.objecter dump_active .. 0 homeless 2024-03-29T11:23:30.299 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:30.320+0000 7f69ff7fe640 10 monclient: get_auth_request con 0x7f69e006db80 auth_method 0 2024-03-29T11:23:30.299 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:30.320+0000 7f69f7fff640 10 client.4345.objecter ms_handle_connect 0x7f69e006db80 2024-03-29T11:23:30.305 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:30.326+0000 7f69f7fff640 1 -- 172.21.15.84:0/2834156928 <== mon.1 v1:172.21.15.179:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f69f0050770 con 0x7f6a00132560 2024-03-29T11:23:30.305 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:30.326+0000 7f69f7fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:23:30.305 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:30.326+0000 7f69f7fff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:23:30.630 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:30.651+0000 7f6a05c4e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:23:30.631 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:30.651+0000 7f6a05c4e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:23:30.631 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:30.651+0000 7f6a05c4e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-29T11:23:30.631 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:30.651+0000 7f6a05c4e640 1 -- 172.21.15.84:0/2834156928 --> v1:172.21.15.179:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6a00135a90 con 0x7f6a00132560 2024-03-29T11:23:30.633 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:30.653+0000 7f69f7fff640 1 -- 172.21.15.84:0/2834156928 <== mon.1 v1:172.21.15.179:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4209477549) 0x7f69f00526a0 con 0x7f6a00132560 2024-03-29T11:23:30.633 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:30.653+0000 7f69f7fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:23:30.633 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:30.653+0000 7f69f7fff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:23:30.634 INFO:teuthology.misc.health.smithi084.stdout: 2024-03-29T11:23:30.635 INFO:teuthology.misc.health.smithi084.stdout:{"epoch":10,"fsid":"2c3e002f-9eb0-4860-853b-4ef1ef562213","created":"2024-03-29T11:19:32.029169+0000","modified":"2024-03-29T11:19:34.200299+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:34.200299+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"5c39e276-2034-4716-bbc0-57a634eefa92","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"cd580e99-109f-4bf1-88c0-a5987bc8b9ed","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"11e60a5a-051d-453f-a190-66387cb35592","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"2c329fbe-bbc4-4030-bf87-d711cad04679","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f4213d4a-bd36-4433-91ae-d7e2b02481d3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"f0cf44ea-6055-48af-805b-aed858102ff9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"d895eaa0-cc0b-4ee4-9b5a-2dfcd89a60b7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"9bc5a7bc-8f52-4db9-96d6-de5e68d52eaa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:23:30.639 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:30.659+0000 7f6a05c4e640 10 librados: watch_flush enter 2024-03-29T11:23:30.639 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:30.659+0000 7f6a05c4e640 10 librados: watch_flush exit 2024-03-29T11:23:30.639 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:30.659+0000 7f6a05c4e640 20 client.4345.objecter shutdown clearing up homeless session... 2024-03-29T11:23:30.639 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:30.659+0000 7f6a05c4e640 10 client.4345.objecter successfully canceled tick 2024-03-29T11:23:30.639 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:30.660+0000 7f6a05c4e640 1 -- 172.21.15.84:0/2834156928 >> v1:172.21.15.179:6816/2981134980 conn(0x7f69e006db80 legacy=0x7f69e0070040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:30.639 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:30.660+0000 7f6a05c4e640 10 monclient: shutdown 2024-03-29T11:23:30.640 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:30.660+0000 7f6a05c4e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:23:30.640 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:30.660+0000 7f6a05c4e640 1 -- 172.21.15.84:0/2834156928 >> v1:172.21.15.179:6789/0 conn(0x7f6a00132560 legacy=0x7f6a001351f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:30.641 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:30.661+0000 7f6a05c4e640 1 -- 172.21.15.84:0/2834156928 shutdown_connections 2024-03-29T11:23:30.641 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:30.662+0000 7f6a05c4e640 1 -- 172.21.15.84:0/2834156928 >> 172.21.15.84:0/2834156928 conn(0x7f6a0006b660 msgr2=0x7f6a00069b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:23:30.642 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:30.662+0000 7f6a05c4e640 1 -- 172.21.15.84:0/2834156928 shutdown_connections 2024-03-29T11:23:30.642 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:30.662+0000 7f6a05c4e640 1 -- 172.21.15.84:0/2834156928 wait complete. 2024-03-29T11:23:30.642 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:30.662+0000 7f6a05c4e640 1 librados: shutdown 2024-03-29T11:23:30.658 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:23:37.464 DEBUG:teuthology.orchestra.run.smithi084:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:23:37.615 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:37.636+0000 7f080cbab640 10 monclient: get_monmap_and_config 2024-03-29T11:23:37.616 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:37.636+0000 7f080cbab640 10 monclient: build_initial_monmap 2024-03-29T11:23:37.616 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:37.636+0000 7f080cbab640 10 monclient: monmap: 2024-03-29T11:23:37.616 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-29T11:23:37.616 INFO:teuthology.misc.health.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:23:37.616 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T11:23:37.637219+0000 2024-03-29T11:23:37.616 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T11:23:37.637219+0000 2024-03-29T11:23:37.616 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-29T11:23:37.616 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:23:37.616 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-29T11:23:37.616 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.noname-c 2024-03-29T11:23:37.616 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-b 2024-03-29T11:23:37.616 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:23:37.619 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:37.637+0000 7f080cbab640 1 Processor -- start 2024-03-29T11:23:37.619 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:37.638+0000 7f080cbab640 1 -- start start 2024-03-29T11:23:37.619 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:37.638+0000 7f080cbab640 10 monclient: init 2024-03-29T11:23:37.619 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:37.638+0000 7f080cbab640 10 monclient: _reopen_session rank -1 2024-03-29T11:23:37.619 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:37.638+0000 7f080cbab640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:23:37.619 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:37.638+0000 7f080cbab640 10 monclient(hunting): picked mon.noname-b con 0x7f08080b1f60 addr v1:172.21.15.84:6790/0 2024-03-29T11:23:37.619 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:37.638+0000 7f080cbab640 10 monclient(hunting): picked mon.noname-a con 0x7f08081439c0 addr v1:172.21.15.84:6789/0 2024-03-29T11:23:37.623 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:37.638+0000 7f080cbab640 10 monclient(hunting): picked mon.noname-c con 0x7f08080b3090 addr v1:172.21.15.179:6789/0 2024-03-29T11:23:37.623 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:37.638+0000 7f080cbab640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f080806ed30 con 0x7f08081439c0 2024-03-29T11:23:37.623 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:37.638+0000 7f080cbab640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f080806f010 con 0x7f08080b3090 2024-03-29T11:23:37.623 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:37.638+0000 7f080cbab640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f08080b3600 con 0x7f08080b1f60 2024-03-29T11:23:37.623 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:37.638+0000 7f080cbab640 10 monclient(hunting): _renew_subs 2024-03-29T11:23:37.623 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:37.638+0000 7f080cbab640 10 monclient(hunting): authenticate will time out at 1078.933017s 2024-03-29T11:23:37.623 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:37.641+0000 7f07fffff640 1 --1- >> v1:172.21.15.84:6789/0 conn(0x7f08081439c0 0x7f08081462a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.84:6789/0 says I am v1:172.21.15.84:49695/0 (socket says 172.21.15.84:49695) 2024-03-29T11:23:37.623 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:37.641+0000 7f07fffff640 1 -- 172.21.15.84:0/545493866 learned_addr learned my addr 172.21.15.84:0/545493866 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-29T11:23:37.627 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:37.644+0000 7f0806ffd640 1 -- 172.21.15.84:0/545493866 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1258024236 0 0) 0x7f08080b3600 con 0x7f08080b1f60 2024-03-29T11:23:37.628 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:37.644+0000 7f0806ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:23:37.628 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:37.644+0000 7f0806ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:23:37.628 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:37.644+0000 7f0806ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:23:37.628 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:37.644+0000 7f0806ffd640 10 monclient(hunting): my global_id is 4385 2024-03-29T11:23:37.628 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:37.644+0000 7f0806ffd640 1 -- 172.21.15.84:0/545493866 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f07f4003520 con 0x7f08080b1f60 2024-03-29T11:23:37.628 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:37.644+0000 7f0806ffd640 1 -- 172.21.15.84:0/545493866 <== mon.0 v1:172.21.15.84:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3301290494 0 0) 0x7f080806ed30 con 0x7f08081439c0 2024-03-29T11:23:37.628 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:37.644+0000 7f0806ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:23:37.628 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:37.644+0000 7f0806ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:23:37.628 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:37.644+0000 7f0806ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:23:37.628 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:37.644+0000 7f0806ffd640 10 monclient(hunting): my global_id is 4389 2024-03-29T11:23:37.628 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:37.644+0000 7f0806ffd640 1 -- 172.21.15.84:0/545493866 --> v1:172.21.15.84:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f08080b3600 con 0x7f08081439c0 2024-03-29T11:23:37.628 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:37.644+0000 7f0806ffd640 1 -- 172.21.15.84:0/545493866 <== mon.2 v1:172.21.15.84:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2468101237 0 0) 0x7f07f4003520 con 0x7f08080b1f60 2024-03-29T11:23:37.628 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:37.644+0000 7f0806ffd640 1 -- 172.21.15.84:0/545493866 >> v1:172.21.15.179:6789/0 conn(0x7f08080b3090 legacy=0x7f0808151bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:23:37.629 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:37.644+0000 7f0806ffd640 1 -- 172.21.15.84:0/545493866 >> v1:172.21.15.84:6789/0 conn(0x7f08081439c0 legacy=0x7f08081462a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:37.629 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:37.644+0000 7f0806ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:23:37.629 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:37.644+0000 7f0806ffd640 1 monclient: found mon.noname-b 2024-03-29T11:23:37.629 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:37.644+0000 7f0806ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:23:37.629 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:37.644+0000 7f0806ffd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.84:6790/0 2024-03-29T11:23:37.629 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:37.644+0000 7f0806ffd640 1 -- 172.21.15.84:0/545493866 --> v1:172.21.15.84:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f080806de90 con 0x7f08080b1f60 2024-03-29T11:23:37.629 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:37.644+0000 7f0806ffd640 10 monclient: _finish_auth 0 2024-03-29T11:23:37.629 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:37.644+0000 7f0806ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:23:37.629 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:37.644+0000 7f0806ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:23:37.629 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:37.644+0000 7f0806ffd640 1 -- 172.21.15.84:0/545493866 <== mon.2 v1:172.21.15.84:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4236868694 0 0) 0x7f07f0001690 con 0x7f08080b1f60 2024-03-29T11:23:37.629 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:37.644+0000 7f0806ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:23:37.629 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:37.644+0000 7f0806ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:23:37.629 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:37.644+0000 7f0806ffd640 10 monclient: dump: 2024-03-29T11:23:37.629 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T11:23:37.629 INFO:teuthology.misc.health.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:23:37.630 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T11:18:53.864025+0000 2024-03-29T11:23:37.630 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T11:18:53.864025+0000 2024-03-29T11:23:37.630 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T11:23:37.630 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:23:37.630 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T11:23:37.630 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T11:23:37.630 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T11:23:37.630 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:23:37.630 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:37.645+0000 7f0806ffd640 1 -- 172.21.15.84:0/545493866 <== mon.2 v1:172.21.15.84:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f07f00017f0 con 0x7f08080b1f60 2024-03-29T11:23:37.630 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:37.645+0000 7f0806ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:23:37.630 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:37.645+0000 7f0806ffd640 1 -- 172.21.15.84:0/545493866 <== mon.2 v1:172.21.15.84:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4236868694 0 0) 0x7f07f0001c00 con 0x7f08080b1f60 2024-03-29T11:23:37.630 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:37.645+0000 7f0806ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:23:37.630 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:37.645+0000 7f0806ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:23:37.630 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:37.645+0000 7f0806ffd640 10 monclient: dump: 2024-03-29T11:23:37.631 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T11:23:37.631 INFO:teuthology.misc.health.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:23:37.631 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T11:18:53.864025+0000 2024-03-29T11:23:37.631 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T11:18:53.864025+0000 2024-03-29T11:23:37.631 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T11:23:37.631 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:23:37.631 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T11:23:37.631 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T11:23:37.631 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T11:23:37.631 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:23:37.631 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:37.645+0000 7f080cbab640 5 monclient: authenticate success, global_id 4385 2024-03-29T11:23:37.631 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:37.645+0000 7f080cbab640 10 monclient: get_monmap_and_config success 2024-03-29T11:23:37.631 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:37.645+0000 7f080cbab640 4 set_mon_vals no callback set 2024-03-29T11:23:37.631 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:37.646+0000 7f080cbab640 10 monclient: shutdown 2024-03-29T11:23:37.632 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:37.646+0000 7f080cbab640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:23:37.632 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:37.646+0000 7f080cbab640 1 -- 172.21.15.84:0/545493866 >> v1:172.21.15.84:6790/0 conn(0x7f08080b1f60 legacy=0x7f08081432b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:37.632 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:37.647+0000 7f080cbab640 1 -- 172.21.15.84:0/545493866 shutdown_connections 2024-03-29T11:23:37.632 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:37.647+0000 7f080cbab640 1 -- 172.21.15.84:0/545493866 >> 172.21.15.84:0/545493866 conn(0x7f080806d160 msgr2=0x7f0808066a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:23:37.632 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:37.648+0000 7f080cbab640 1 -- 172.21.15.84:0/545493866 shutdown_connections 2024-03-29T11:23:37.636 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:37.650+0000 7f080cbab640 1 -- 172.21.15.84:0/545493866 wait complete. 2024-03-29T11:23:37.636 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:37.650+0000 7f080cbab640 10 monclient: build_initial_monmap 2024-03-29T11:23:37.636 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:37.650+0000 7f080cbab640 10 monclient: monmap: 2024-03-29T11:23:37.636 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-29T11:23:37.636 INFO:teuthology.misc.health.smithi084.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:23:37.636 INFO:teuthology.misc.health.smithi084.stderr:last_changed 0.000000 2024-03-29T11:23:37.636 INFO:teuthology.misc.health.smithi084.stderr:created 0.000000 2024-03-29T11:23:37.636 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-29T11:23:37.636 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:23:37.636 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-29T11:23:37.636 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.noname-b 2024-03-29T11:23:37.637 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-c 2024-03-29T11:23:37.637 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:23:37.637 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:37.651+0000 7f080cbab640 1 librados: starting msgr at 2024-03-29T11:23:37.637 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:37.651+0000 7f080cbab640 1 librados: starting objecter 2024-03-29T11:23:37.637 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:37.651+0000 7f080cbab640 1 Processor -- start 2024-03-29T11:23:37.637 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:37.651+0000 7f080cbab640 1 -- start start 2024-03-29T11:23:37.637 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:37.651+0000 7f080cbab640 1 librados: setting wanted keys 2024-03-29T11:23:37.637 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:37.651+0000 7f080cbab640 1 librados: calling monclient init 2024-03-29T11:23:37.637 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:37.651+0000 7f080cbab640 10 monclient: init 2024-03-29T11:23:37.637 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:37.652+0000 7f080cbab640 10 monclient: _reopen_session rank -1 2024-03-29T11:23:37.637 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:37.652+0000 7f080cbab640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:23:37.637 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:37.652+0000 7f080cbab640 10 monclient(hunting): picked mon.noname-b con 0x7f08080b3090 addr v1:172.21.15.179:6789/0 2024-03-29T11:23:37.637 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:37.655+0000 7f080cbab640 10 monclient(hunting): picked mon.noname-a con 0x7f08081439c0 addr v1:172.21.15.84:6789/0 2024-03-29T11:23:37.637 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:37.655+0000 7f080cbab640 10 monclient(hunting): picked mon.noname-c con 0x7f08080b86d0 addr v1:172.21.15.84:6790/0 2024-03-29T11:23:37.637 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:37.655+0000 7f080cbab640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f08080b6390 con 0x7f08081439c0 2024-03-29T11:23:37.638 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:37.655+0000 7f080cbab640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f08080b82b0 con 0x7f08080b3090 2024-03-29T11:23:37.638 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:37.655+0000 7f080cbab640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0808202b60 con 0x7f08080b86d0 2024-03-29T11:23:37.638 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:37.655+0000 7f080cbab640 10 monclient(hunting): _renew_subs 2024-03-29T11:23:37.638 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:37.655+0000 7f080cbab640 10 monclient(hunting): authenticate will time out at 1078.949811s 2024-03-29T11:23:37.638 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:37.655+0000 7f07fffff640 1 --1- >> v1:172.21.15.84:6789/0 conn(0x7f08081439c0 0x7f0808137390 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.84:6789/0 says I am v1:172.21.15.84:49903/0 (socket says 172.21.15.84:49903) 2024-03-29T11:23:37.638 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:37.655+0000 7f07fffff640 1 -- 172.21.15.84:0/2368417119 learned_addr learned my addr 172.21.15.84:0/2368417119 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-29T11:23:37.638 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:37.656+0000 7f0804ff9640 10 client.?.objecter ms_handle_connect 0x7f08081439c0 2024-03-29T11:23:37.638 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:37.656+0000 7f0804ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:23:37.638 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:37.656+0000 7f0804ff9640 1 -- 172.21.15.84:0/2368417119 <== mon.0 v1:172.21.15.84:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2084498633 0 0) 0x7f08080b6390 con 0x7f08081439c0 2024-03-29T11:23:37.638 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:37.656+0000 7f0804ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:23:37.638 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:37.656+0000 7f0804ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:23:37.638 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:37.656+0000 7f0804ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:23:37.638 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:37.656+0000 7f0804ff9640 10 monclient(hunting): my global_id is 4392 2024-03-29T11:23:37.639 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:37.656+0000 7f0804ff9640 1 -- 172.21.15.84:0/2368417119 --> v1:172.21.15.84:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f07e8004eb0 con 0x7f08081439c0 2024-03-29T11:23:37.639 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:37.656+0000 7f0804ff9640 1 -- 172.21.15.84:0/2368417119 <== mon.0 v1:172.21.15.84:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2004749008 0 0) 0x7f07e8004eb0 con 0x7f08081439c0 2024-03-29T11:23:37.639 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:37.656+0000 7f0804ff9640 1 -- 172.21.15.84:0/2368417119 --> v1:172.21.15.84:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f08080b6390 con 0x7f08081439c0 2024-03-29T11:23:37.639 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:37.656+0000 7f0804ff9640 1 -- 172.21.15.84:0/2368417119 <== mon.0 v1:172.21.15.84:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4236868694 0 0) 0x7f07f8001910 con 0x7f08081439c0 2024-03-29T11:23:37.639 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:37.656+0000 7f0804ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:23:37.639 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:37.656+0000 7f0804ff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:23:37.639 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:37.656+0000 7f0804ff9640 10 monclient(hunting): dump: 2024-03-29T11:23:37.639 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T11:23:37.639 INFO:teuthology.misc.health.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:23:37.639 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T11:18:53.864025+0000 2024-03-29T11:23:37.639 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T11:18:53.864025+0000 2024-03-29T11:23:37.639 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T11:23:37.639 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:23:37.639 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T11:23:37.639 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T11:23:37.640 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T11:23:37.640 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:23:37.640 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:37.656+0000 7f0804ff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:23:37.642 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:37.656+0000 7f0804ff9640 1 -- 172.21.15.84:0/2368417119 <== mon.0 v1:172.21.15.84:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2928871311 0 0) 0x7f08080b6390 con 0x7f08081439c0 2024-03-29T11:23:37.642 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:37.657+0000 7f0804ff9640 1 -- 172.21.15.84:0/2368417119 >> v1:172.21.15.84:6790/0 conn(0x7f08080b86d0 legacy=0x7f0808137aa0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:23:37.642 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:37.660+0000 7f0804ff9640 1 -- 172.21.15.84:0/2368417119 >> v1:172.21.15.179:6789/0 conn(0x7f08080b3090 legacy=0x7f08080b6a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:37.642 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:37.660+0000 7f0804ff9640 10 monclient: _finish_hunting 0 2024-03-29T11:23:37.642 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:37.660+0000 7f0804ff9640 1 monclient: found mon.a 2024-03-29T11:23:37.642 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:37.660+0000 7f0804ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:23:37.643 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:37.660+0000 7f0804ff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.84:6789/0 2024-03-29T11:23:37.643 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:37.660+0000 7f0804ff9640 1 -- 172.21.15.84:0/2368417119 --> v1:172.21.15.84:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0808203d00 con 0x7f08081439c0 2024-03-29T11:23:37.643 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:37.660+0000 7f0804ff9640 10 monclient: _finish_auth 0 2024-03-29T11:23:37.643 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:37.660+0000 7f0804ff9640 10 monclient: _check_auth_tickets 2024-03-29T11:23:37.643 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:37.660+0000 7f0804ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:23:37.643 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:37.660+0000 7f080cbab640 5 monclient: authenticate success, global_id 4392 2024-03-29T11:23:37.643 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:37.660+0000 7f080cbab640 10 monclient: _renew_subs 2024-03-29T11:23:37.643 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:37.660+0000 7f080cbab640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.84:6789/0 2024-03-29T11:23:37.643 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:37.660+0000 7f080cbab640 1 -- 172.21.15.84:0/2368417119 --> v1:172.21.15.84:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0808202cf0 con 0x7f08081439c0 2024-03-29T11:23:37.643 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:37.660+0000 7f080cbab640 10 client.4392.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:23:37.643 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:37.660+0000 7f080cbab640 10 monclient: _renew_subs 2024-03-29T11:23:37.643 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:37.660+0000 7f080cbab640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.84:6789/0 2024-03-29T11:23:37.643 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:37.660+0000 7f080cbab640 1 -- 172.21.15.84:0/2368417119 --> v1:172.21.15.84:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f08082031d0 con 0x7f08081439c0 2024-03-29T11:23:37.643 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:37.660+0000 7f080cbab640 1 librados: init done 2024-03-29T11:23:37.643 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:37.660+0000 7f0804ff9640 10 client.4392.objecter ms_handle_connect 0x7f08080b3090 2024-03-29T11:23:37.644 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:37.660+0000 7f0804ff9640 10 client.4392.objecter resend_mon_ops 2024-03-29T11:23:37.648 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:37.666+0000 7f0804ff9640 1 -- 172.21.15.84:0/2368417119 <== mon.0 v1:172.21.15.84:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f07f8001a70 con 0x7f08081439c0 2024-03-29T11:23:37.648 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:37.666+0000 7f0804ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:23:37.648 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:37.666+0000 7f0804ff9640 1 -- 172.21.15.84:0/2368417119 <== mon.0 v1:172.21.15.84:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4236868694 0 0) 0x7f07f8004a90 con 0x7f08081439c0 2024-03-29T11:23:37.648 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:37.666+0000 7f0804ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:23:37.648 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:37.666+0000 7f0804ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:23:37.648 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:37.666+0000 7f0804ff9640 10 monclient: dump: 2024-03-29T11:23:37.648 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T11:23:37.648 INFO:teuthology.misc.health.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:23:37.648 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T11:18:53.864025+0000 2024-03-29T11:23:37.648 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T11:18:53.864025+0000 2024-03-29T11:23:37.648 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T11:23:37.648 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:23:37.648 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T11:23:37.648 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T11:23:37.649 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T11:23:37.649 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:23:37.649 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:37.666+0000 7f0804ff9640 1 -- 172.21.15.84:0/2368417119 <== mon.0 v1:172.21.15.84:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4033942909 0 0) 0x7f07f801ab20 con 0x7f08081439c0 2024-03-29T11:23:37.649 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:37.667+0000 7f0804ff9640 1 -- 172.21.15.84:0/2368417119 <== mon.0 v1:172.21.15.84:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2808139587 0 0) 0x7f07f8084c10 con 0x7f08081439c0 2024-03-29T11:23:37.649 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:37.667+0000 7f0804ff9640 10 client.4392.objecter ms_dispatch 0x7f0808000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:23:37.649 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:37.667+0000 7f0804ff9640 3 client.4392.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:23:37.649 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:37.667+0000 7f0804ff9640 3 client.4392.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:23:37.649 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:37.667+0000 7f0804ff9640 20 client.4392.objecter dump_active .. 0 homeless 2024-03-29T11:23:37.649 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:37.667+0000 7f08077fe640 10 monclient: get_auth_request con 0x7f07e806dd10 auth_method 0 2024-03-29T11:23:37.649 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:37.668+0000 7f0804ff9640 10 client.4392.objecter ms_handle_connect 0x7f07e806dd10 2024-03-29T11:23:37.649 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:37.668+0000 7f08067fc640 4 set_mon_vals no callback set 2024-03-29T11:23:37.651 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:37.672+0000 7f080cbab640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:23:37.651 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:37.672+0000 7f080cbab640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:23:37.651 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:37.672+0000 7f080cbab640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.84:6789/0 2024-03-29T11:23:37.651 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:37.672+0000 7f080cbab640 1 -- 172.21.15.84:0/2368417119 --> v1:172.21.15.84:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f07ec005080 con 0x7f08081439c0 2024-03-29T11:23:37.662 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:37.682+0000 7f0804ff9640 1 -- 172.21.15.84:0/2368417119 <== mon.0 v1:172.21.15.84:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f07f80503a0 con 0x7f08081439c0 2024-03-29T11:23:37.662 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:37.682+0000 7f0804ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:23:37.662 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:37.682+0000 7f0804ff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:23:38.001 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:38.021+0000 7f080cbab640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:23:38.001 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:38.021+0000 7f080cbab640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:23:38.001 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:38.021+0000 7f080cbab640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.84:6789/0 2024-03-29T11:23:38.001 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:38.021+0000 7f080cbab640 1 -- 172.21.15.84:0/2368417119 --> v1:172.21.15.84:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f07ec005330 con 0x7f08081439c0 2024-03-29T11:23:38.002 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:38.022+0000 7f0804ff9640 1 -- 172.21.15.84:0/2368417119 <== mon.0 v1:172.21.15.84:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4209477549) 0x7f07f80524a0 con 0x7f08081439c0 2024-03-29T11:23:38.002 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:38.022+0000 7f0804ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:23:38.002 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:38.022+0000 7f0804ff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:23:38.002 INFO:teuthology.misc.health.smithi084.stdout: 2024-03-29T11:23:38.002 INFO:teuthology.misc.health.smithi084.stdout:{"epoch":10,"fsid":"2c3e002f-9eb0-4860-853b-4ef1ef562213","created":"2024-03-29T11:19:32.029169+0000","modified":"2024-03-29T11:19:34.200299+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:34.200299+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"5c39e276-2034-4716-bbc0-57a634eefa92","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"cd580e99-109f-4bf1-88c0-a5987bc8b9ed","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"11e60a5a-051d-453f-a190-66387cb35592","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"2c329fbe-bbc4-4030-bf87-d711cad04679","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f4213d4a-bd36-4433-91ae-d7e2b02481d3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"f0cf44ea-6055-48af-805b-aed858102ff9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"d895eaa0-cc0b-4ee4-9b5a-2dfcd89a60b7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"9bc5a7bc-8f52-4db9-96d6-de5e68d52eaa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:23:38.006 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:38.026+0000 7f080cbab640 10 librados: watch_flush enter 2024-03-29T11:23:38.006 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:38.026+0000 7f080cbab640 10 librados: watch_flush exit 2024-03-29T11:23:38.006 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:38.026+0000 7f080cbab640 20 client.4392.objecter shutdown clearing up homeless session... 2024-03-29T11:23:38.006 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:38.026+0000 7f080cbab640 10 client.4392.objecter successfully canceled tick 2024-03-29T11:23:38.006 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:38.026+0000 7f080cbab640 1 -- 172.21.15.84:0/2368417119 >> v1:172.21.15.179:6816/2981134980 conn(0x7f07e806dd10 legacy=0x7f07e80701d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:38.006 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:38.026+0000 7f080cbab640 10 monclient: shutdown 2024-03-29T11:23:38.006 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:38.026+0000 7f080cbab640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:23:38.006 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:38.026+0000 7f080cbab640 1 -- 172.21.15.84:0/2368417119 >> v1:172.21.15.84:6789/0 conn(0x7f08081439c0 legacy=0x7f0808137390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:38.006 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:38.026+0000 7f080cbab640 1 -- 172.21.15.84:0/2368417119 shutdown_connections 2024-03-29T11:23:38.006 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:38.026+0000 7f080cbab640 1 -- 172.21.15.84:0/2368417119 >> 172.21.15.84:0/2368417119 conn(0x7f080806d160 msgr2=0x7f0808071040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:23:38.007 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:38.026+0000 7f080cbab640 1 -- 172.21.15.84:0/2368417119 shutdown_connections 2024-03-29T11:23:38.007 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:38.026+0000 7f080cbab640 1 -- 172.21.15.84:0/2368417119 wait complete. 2024-03-29T11:23:38.007 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:38.027+0000 7f080cbab640 1 librados: shutdown 2024-03-29T11:23:38.023 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:23:44.830 DEBUG:teuthology.orchestra.run.smithi084:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:23:44.995 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:45.015+0000 7fc761a16640 10 monclient: get_monmap_and_config 2024-03-29T11:23:44.995 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:45.015+0000 7fc761a16640 10 monclient: build_initial_monmap 2024-03-29T11:23:44.995 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:45.015+0000 7fc761a16640 10 monclient: monmap: 2024-03-29T11:23:44.995 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-29T11:23:44.995 INFO:teuthology.misc.health.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:23:44.995 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T11:23:45.016109+0000 2024-03-29T11:23:44.995 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T11:23:45.016109+0000 2024-03-29T11:23:44.995 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-29T11:23:44.995 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:23:44.995 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-29T11:23:44.996 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.noname-c 2024-03-29T11:23:44.996 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-b 2024-03-29T11:23:44.996 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:23:44.996 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:45.016+0000 7fc761a16640 1 Processor -- start 2024-03-29T11:23:44.996 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:45.016+0000 7fc761a16640 1 -- start start 2024-03-29T11:23:44.996 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:45.016+0000 7fc761a16640 10 monclient: init 2024-03-29T11:23:45.003 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:45.017+0000 7fc761a16640 10 monclient: _reopen_session rank -1 2024-03-29T11:23:45.003 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:45.017+0000 7fc761a16640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:23:45.003 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:45.017+0000 7fc761a16640 10 monclient(hunting): picked mon.noname-c con 0x7fc75c0b0860 addr v1:172.21.15.179:6789/0 2024-03-29T11:23:45.003 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:45.021+0000 7fc761a16640 10 monclient(hunting): picked mon.noname-a con 0x7fc75c072450 addr v1:172.21.15.84:6789/0 2024-03-29T11:23:45.003 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:45.021+0000 7fc761a16640 10 monclient(hunting): picked mon.noname-b con 0x7fc75c13ee40 addr v1:172.21.15.84:6790/0 2024-03-29T11:23:45.003 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:45.021+0000 7fc761a16640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc75c0ad840 con 0x7fc75c072450 2024-03-29T11:23:45.003 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:45.021+0000 7fc761a16640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc75c13c830 con 0x7fc75c0b0860 2024-03-29T11:23:45.003 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:45.021+0000 7fc761a16640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc75c13c9c0 con 0x7fc75c13ee40 2024-03-29T11:23:45.004 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:45.021+0000 7fc761a16640 10 monclient(hunting): _renew_subs 2024-03-29T11:23:45.004 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:45.021+0000 7fc761a16640 10 monclient(hunting): authenticate will time out at 1086.315445s 2024-03-29T11:23:45.004 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:45.021+0000 7fc75a7fc640 1 --1- >> v1:172.21.15.84:6789/0 conn(0x7fc75c072450 0x7fc75c13c120 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.84:6789/0 says I am v1:172.21.15.84:41375/0 (socket says 172.21.15.84:41375) 2024-03-29T11:23:45.004 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:45.021+0000 7fc75a7fc640 1 -- 172.21.15.84:0/1984623317 learned_addr learned my addr 172.21.15.84:0/1984623317 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-29T11:23:45.006 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:45.026+0000 7fc759ffb640 1 -- 172.21.15.84:0/1984623317 <== mon.0 v1:172.21.15.84:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2770939049 0 0) 0x7fc75c0ad840 con 0x7fc75c072450 2024-03-29T11:23:45.006 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:45.026+0000 7fc759ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:23:45.006 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:45.026+0000 7fc759ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:23:45.006 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:45.026+0000 7fc759ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:23:45.006 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:45.026+0000 7fc759ffb640 10 monclient(hunting): my global_id is 4395 2024-03-29T11:23:45.006 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:45.026+0000 7fc759ffb640 1 -- 172.21.15.84:0/1984623317 --> v1:172.21.15.84:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc744005730 con 0x7fc75c072450 2024-03-29T11:23:45.009 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:45.028+0000 7fc759ffb640 1 -- 172.21.15.84:0/1984623317 <== mon.1 v1:172.21.15.179:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3649705497 0 0) 0x7fc75c13c830 con 0x7fc75c0b0860 2024-03-29T11:23:45.009 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:45.028+0000 7fc759ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:23:45.009 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:45.028+0000 7fc759ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:23:45.009 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:45.028+0000 7fc759ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:23:45.009 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:45.028+0000 7fc759ffb640 10 monclient(hunting): my global_id is 4348 2024-03-29T11:23:45.010 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:45.028+0000 7fc759ffb640 1 -- 172.21.15.84:0/1984623317 --> v1:172.21.15.179:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc75c0ad840 con 0x7fc75c0b0860 2024-03-29T11:23:45.010 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:45.029+0000 7fc759ffb640 1 -- 172.21.15.84:0/1984623317 <== mon.1 v1:172.21.15.179:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2855451934 0 0) 0x7fc75c0ad840 con 0x7fc75c0b0860 2024-03-29T11:23:45.010 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:45.029+0000 7fc759ffb640 1 -- 172.21.15.84:0/1984623317 >> v1:172.21.15.84:6790/0 conn(0x7fc75c13ee40 legacy=0x7fc75c13f260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:23:45.010 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:45.029+0000 7fc759ffb640 1 -- 172.21.15.84:0/1984623317 >> v1:172.21.15.84:6789/0 conn(0x7fc75c072450 legacy=0x7fc75c13c120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:45.010 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:45.029+0000 7fc759ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:23:45.010 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:45.029+0000 7fc759ffb640 1 monclient: found mon.noname-c 2024-03-29T11:23:45.010 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:45.029+0000 7fc759ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:23:45.010 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:45.029+0000 7fc759ffb640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.179:6789/0 2024-03-29T11:23:45.010 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:45.029+0000 7fc759ffb640 1 -- 172.21.15.84:0/1984623317 --> v1:172.21.15.179:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc75c13f970 con 0x7fc75c0b0860 2024-03-29T11:23:45.010 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:45.029+0000 7fc759ffb640 10 monclient: _finish_auth 0 2024-03-29T11:23:45.010 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:45.029+0000 7fc759ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:23:45.010 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:45.029+0000 7fc759ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:23:45.011 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:45.029+0000 7fc761a16640 5 monclient: authenticate success, global_id 4348 2024-03-29T11:23:45.011 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:45.029+0000 7fc761a16640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:23:45.011 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:45.029+0000 7fc759ffb640 1 -- 172.21.15.84:0/1984623317 <== mon.1 v1:172.21.15.179:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4236868694 0 0) 0x7fc7540019b0 con 0x7fc75c0b0860 2024-03-29T11:23:45.011 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:45.029+0000 7fc759ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:23:45.011 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:45.029+0000 7fc759ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:23:45.011 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:45.029+0000 7fc759ffb640 10 monclient: dump: 2024-03-29T11:23:45.011 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T11:23:45.011 INFO:teuthology.misc.health.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:23:45.011 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T11:18:53.864025+0000 2024-03-29T11:23:45.011 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T11:18:53.864025+0000 2024-03-29T11:23:45.011 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T11:23:45.011 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:23:45.011 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T11:23:45.012 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T11:23:45.012 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T11:23:45.012 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:23:45.012 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:45.029+0000 7fc761a16640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:23:45.013 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:45.030+0000 7fc759ffb640 1 -- 172.21.15.84:0/1984623317 <== mon.1 v1:172.21.15.179:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc754001c00 con 0x7fc75c0b0860 2024-03-29T11:23:45.013 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:45.030+0000 7fc759ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:23:45.013 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:45.030+0000 7fc761a16640 10 monclient: get_monmap_and_config success 2024-03-29T11:23:45.013 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:45.030+0000 7fc761a16640 4 set_mon_vals no callback set 2024-03-29T11:23:45.013 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:45.031+0000 7fc761a16640 10 monclient: shutdown 2024-03-29T11:23:45.013 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:45.031+0000 7fc761a16640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:23:45.013 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:45.031+0000 7fc761a16640 1 -- 172.21.15.84:0/1984623317 >> v1:172.21.15.179:6789/0 conn(0x7fc75c0b0860 legacy=0x7fc75c0acfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:45.013 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:45.031+0000 7fc759ffb640 1 -- 172.21.15.84:0/1984623317 <== mon.1 v1:172.21.15.179:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4236868694 0 0) 0x7fc7540019b0 con 0x7fc75c0b0860 2024-03-29T11:23:45.013 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:45.031+0000 7fc759ffb640 10 monclient: discarding stray monitor message mon_map magic: 0 v1 2024-03-29T11:23:45.013 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:45.031+0000 7fc761a16640 1 -- 172.21.15.84:0/1984623317 shutdown_connections 2024-03-29T11:23:45.013 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:45.031+0000 7fc761a16640 1 -- 172.21.15.84:0/1984623317 >> 172.21.15.84:0/1984623317 conn(0x7fc75c06b660 msgr2=0x7fc75c06ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:23:45.013 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:45.033+0000 7fc761a16640 1 -- 172.21.15.84:0/1984623317 shutdown_connections 2024-03-29T11:23:45.015 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:45.034+0000 7fc761a16640 1 -- 172.21.15.84:0/1984623317 wait complete. 2024-03-29T11:23:45.017 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:45.035+0000 7fc761a16640 10 monclient: build_initial_monmap 2024-03-29T11:23:45.018 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:45.035+0000 7fc761a16640 10 monclient: monmap: 2024-03-29T11:23:45.018 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-29T11:23:45.018 INFO:teuthology.misc.health.smithi084.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:23:45.018 INFO:teuthology.misc.health.smithi084.stderr:last_changed 0.000000 2024-03-29T11:23:45.018 INFO:teuthology.misc.health.smithi084.stderr:created 0.000000 2024-03-29T11:23:45.018 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-29T11:23:45.018 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:23:45.018 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-29T11:23:45.018 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.noname-b 2024-03-29T11:23:45.018 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-c 2024-03-29T11:23:45.018 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:23:45.018 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:45.036+0000 7fc761a16640 1 librados: starting msgr at 2024-03-29T11:23:45.018 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:45.036+0000 7fc761a16640 1 librados: starting objecter 2024-03-29T11:23:45.019 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:45.036+0000 7fc761a16640 1 Processor -- start 2024-03-29T11:23:45.019 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:45.036+0000 7fc761a16640 1 -- start start 2024-03-29T11:23:45.019 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:45.036+0000 7fc761a16640 1 librados: setting wanted keys 2024-03-29T11:23:45.019 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:45.036+0000 7fc761a16640 1 librados: calling monclient init 2024-03-29T11:23:45.019 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:45.036+0000 7fc761a16640 10 monclient: init 2024-03-29T11:23:45.021 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:45.037+0000 7fc761a16640 10 monclient: _reopen_session rank -1 2024-03-29T11:23:45.021 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:45.037+0000 7fc761a16640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:23:45.021 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:45.037+0000 7fc761a16640 10 monclient(hunting): picked mon.noname-c con 0x7fc75c072450 addr v1:172.21.15.84:6790/0 2024-03-29T11:23:45.021 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:45.037+0000 7fc761a16640 10 monclient(hunting): picked mon.noname-a con 0x7fc75c0b0860 addr v1:172.21.15.84:6789/0 2024-03-29T11:23:45.021 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:45.037+0000 7fc761a16640 10 monclient(hunting): picked mon.noname-b con 0x7fc75c13ee40 addr v1:172.21.15.179:6789/0 2024-03-29T11:23:45.021 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:45.037+0000 7fc761a16640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc75c141300 con 0x7fc75c0b0860 2024-03-29T11:23:45.021 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:45.037+0000 7fc761a16640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc75c1322d0 con 0x7fc75c13ee40 2024-03-29T11:23:45.022 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:45.037+0000 7fc761a16640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc75c142c40 con 0x7fc75c072450 2024-03-29T11:23:45.022 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:45.037+0000 7fc761a16640 10 monclient(hunting): _renew_subs 2024-03-29T11:23:45.022 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:45.037+0000 7fc761a16640 10 monclient(hunting): authenticate will time out at 1086.331742s 2024-03-29T11:23:45.023 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:45.042+0000 7fc75affd640 1 --1- >> v1:172.21.15.84:6790/0 conn(0x7fc75c072450 0x7fc75c133c50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.84:6790/0 says I am v1:172.21.15.84:54159/0 (socket says 172.21.15.84:54159) 2024-03-29T11:23:45.023 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:45.042+0000 7fc75affd640 1 -- 172.21.15.84:0/3362748996 learned_addr learned my addr 172.21.15.84:0/3362748996 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-29T11:23:45.024 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:45.043+0000 7fc743fff640 10 client.?.objecter ms_handle_connect 0x7fc75c072450 2024-03-29T11:23:45.024 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:45.043+0000 7fc743fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:23:45.024 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:45.043+0000 7fc743fff640 10 client.?.objecter ms_handle_connect 0x7fc75c0b0860 2024-03-29T11:23:45.024 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:45.043+0000 7fc743fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:23:45.024 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:45.043+0000 7fc743fff640 10 client.?.objecter ms_handle_connect 0x7fc75c13ee40 2024-03-29T11:23:45.024 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:45.043+0000 7fc743fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:23:45.024 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:45.043+0000 7fc743fff640 1 -- 172.21.15.84:0/3362748996 <== mon.1 v1:172.21.15.179:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4033088110 0 0) 0x7fc75c1322d0 con 0x7fc75c13ee40 2024-03-29T11:23:45.024 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:45.044+0000 7fc743fff640 10 monclient(hunting): _negotiate 2024-03-29T11:23:45.024 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:45.044+0000 7fc743fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:23:45.024 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:45.044+0000 7fc743fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:23:45.024 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:45.044+0000 7fc743fff640 10 monclient(hunting): my global_id is 4351 2024-03-29T11:23:45.024 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:45.044+0000 7fc743fff640 1 -- 172.21.15.84:0/3362748996 --> v1:172.21.15.179:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc748004eb0 con 0x7fc75c13ee40 2024-03-29T11:23:45.028 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:45.044+0000 7fc743fff640 1 -- 172.21.15.84:0/3362748996 <== mon.1 v1:172.21.15.179:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1020931813 0 0) 0x7fc748004eb0 con 0x7fc75c13ee40 2024-03-29T11:23:45.028 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:45.045+0000 7fc743fff640 1 -- 172.21.15.84:0/3362748996 --> v1:172.21.15.179:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc75c1322d0 con 0x7fc75c13ee40 2024-03-29T11:23:45.029 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:45.045+0000 7fc743fff640 1 -- 172.21.15.84:0/3362748996 <== mon.1 v1:172.21.15.179:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4236868694 0 0) 0x7fc750001680 con 0x7fc75c13ee40 2024-03-29T11:23:45.029 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:45.045+0000 7fc743fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:23:45.029 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:45.045+0000 7fc743fff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:23:45.029 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:45.045+0000 7fc743fff640 10 monclient(hunting): dump: 2024-03-29T11:23:45.029 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T11:23:45.029 INFO:teuthology.misc.health.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:23:45.029 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T11:18:53.864025+0000 2024-03-29T11:23:45.029 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T11:18:53.864025+0000 2024-03-29T11:23:45.029 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T11:23:45.029 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:23:45.029 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T11:23:45.029 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T11:23:45.029 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T11:23:45.030 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:23:45.030 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:45.045+0000 7fc743fff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:23:45.030 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:45.045+0000 7fc743fff640 1 -- 172.21.15.84:0/3362748996 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3883462825 0 0) 0x7fc75c142c40 con 0x7fc75c072450 2024-03-29T11:23:45.030 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:45.045+0000 7fc743fff640 10 monclient(hunting): _negotiate 2024-03-29T11:23:45.030 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:45.045+0000 7fc743fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:23:45.030 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:45.045+0000 7fc743fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:23:45.030 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:45.045+0000 7fc743fff640 10 monclient(hunting): my global_id is 4388 2024-03-29T11:23:45.030 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:45.045+0000 7fc743fff640 1 -- 172.21.15.84:0/3362748996 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc748004eb0 con 0x7fc75c072450 2024-03-29T11:23:45.030 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:45.045+0000 7fc743fff640 1 -- 172.21.15.84:0/3362748996 <== mon.1 v1:172.21.15.179:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2151082383 0 0) 0x7fc75c1322d0 con 0x7fc75c13ee40 2024-03-29T11:23:45.030 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:45.046+0000 7fc743fff640 1 -- 172.21.15.84:0/3362748996 >> v1:172.21.15.84:6790/0 conn(0x7fc75c072450 legacy=0x7fc75c133c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:45.030 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:45.046+0000 7fc743fff640 1 -- 172.21.15.84:0/3362748996 >> v1:172.21.15.84:6789/0 conn(0x7fc75c0b0860 legacy=0x7fc75c1314b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:45.030 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:45.046+0000 7fc743fff640 10 monclient: _finish_hunting 0 2024-03-29T11:23:45.030 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:45.046+0000 7fc743fff640 1 monclient: found mon.b 2024-03-29T11:23:45.030 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:45.046+0000 7fc743fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:23:45.030 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:45.046+0000 7fc743fff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-29T11:23:45.031 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:45.046+0000 7fc743fff640 1 -- 172.21.15.84:0/3362748996 --> v1:172.21.15.179:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc75c142f20 con 0x7fc75c13ee40 2024-03-29T11:23:45.031 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:45.046+0000 7fc743fff640 10 monclient: _finish_auth 0 2024-03-29T11:23:45.031 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:45.046+0000 7fc743fff640 10 monclient: _check_auth_tickets 2024-03-29T11:23:45.031 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:45.046+0000 7fc743fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:23:45.031 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:45.046+0000 7fc761a16640 5 monclient: authenticate success, global_id 4351 2024-03-29T11:23:45.031 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:45.046+0000 7fc761a16640 10 monclient: _renew_subs 2024-03-29T11:23:45.031 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:45.046+0000 7fc761a16640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-29T11:23:45.031 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:45.046+0000 7fc761a16640 1 -- 172.21.15.84:0/3362748996 --> v1:172.21.15.179:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc75c142240 con 0x7fc75c13ee40 2024-03-29T11:23:45.031 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:45.046+0000 7fc761a16640 10 client.4351.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:23:45.031 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:45.046+0000 7fc761a16640 10 monclient: _renew_subs 2024-03-29T11:23:45.031 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:45.046+0000 7fc761a16640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-29T11:23:45.031 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:45.046+0000 7fc761a16640 1 -- 172.21.15.84:0/3362748996 --> v1:172.21.15.179:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc75c141060 con 0x7fc75c13ee40 2024-03-29T11:23:45.031 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:45.046+0000 7fc761a16640 1 librados: init done 2024-03-29T11:23:45.032 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:45.046+0000 7fc743fff640 1 -- 172.21.15.84:0/3362748996 <== mon.1 v1:172.21.15.179:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc750001bd0 con 0x7fc75c13ee40 2024-03-29T11:23:45.032 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:45.046+0000 7fc743fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:23:45.032 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:45.046+0000 7fc743fff640 1 -- 172.21.15.84:0/3362748996 <== mon.1 v1:172.21.15.179:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4236868694 0 0) 0x7fc750003790 con 0x7fc75c13ee40 2024-03-29T11:23:45.032 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:45.046+0000 7fc743fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:23:45.032 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:45.046+0000 7fc743fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:23:45.032 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:45.046+0000 7fc7597fa640 4 set_mon_vals no callback set 2024-03-29T11:23:45.032 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:45.046+0000 7fc743fff640 10 monclient: dump: 2024-03-29T11:23:45.032 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T11:23:45.032 INFO:teuthology.misc.health.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:23:45.032 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T11:18:53.864025+0000 2024-03-29T11:23:45.032 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T11:18:53.864025+0000 2024-03-29T11:23:45.032 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T11:23:45.032 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:23:45.032 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T11:23:45.032 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T11:23:45.033 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T11:23:45.033 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:23:45.034 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:45.050+0000 7fc743fff640 1 -- 172.21.15.84:0/3362748996 <== mon.1 v1:172.21.15.179:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4033942909 0 0) 0x7fc750019760 con 0x7fc75c13ee40 2024-03-29T11:23:45.034 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:45.051+0000 7fc75affd640 10 monclient: get_auth_request con 0x7fc74806dee0 auth_method 0 2024-03-29T11:23:45.034 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:45.051+0000 7fc743fff640 1 -- 172.21.15.84:0/3362748996 <== mon.1 v1:172.21.15.179:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2808139587 0 0) 0x7fc750083600 con 0x7fc75c13ee40 2024-03-29T11:23:45.034 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:45.051+0000 7fc743fff640 10 client.4351.objecter ms_dispatch 0x7fc75c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:23:45.034 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:45.051+0000 7fc743fff640 3 client.4351.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:23:45.034 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:45.051+0000 7fc743fff640 3 client.4351.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:23:45.034 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:45.051+0000 7fc743fff640 20 client.4351.objecter dump_active .. 0 homeless 2024-03-29T11:23:45.034 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:45.052+0000 7fc743fff640 10 client.4351.objecter ms_handle_connect 0x7fc74806dee0 2024-03-29T11:23:45.035 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:45.053+0000 7fc761a16640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:23:45.035 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:45.053+0000 7fc761a16640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:23:45.035 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:45.053+0000 7fc761a16640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-29T11:23:45.035 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:45.053+0000 7fc761a16640 1 -- 172.21.15.84:0/3362748996 --> v1:172.21.15.179:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc73c005080 con 0x7fc75c13ee40 2024-03-29T11:23:45.038 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:45.058+0000 7fc743fff640 1 -- 172.21.15.84:0/3362748996 <== mon.1 v1:172.21.15.179:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fc75004eee0 con 0x7fc75c13ee40 2024-03-29T11:23:45.038 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:45.058+0000 7fc743fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:23:45.038 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:45.058+0000 7fc743fff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:23:45.372 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:45.392+0000 7fc761a16640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:23:45.372 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:45.392+0000 7fc761a16640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:23:45.372 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:45.392+0000 7fc761a16640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-29T11:23:45.372 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:45.392+0000 7fc761a16640 1 -- 172.21.15.84:0/3362748996 --> v1:172.21.15.179:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc73c005330 con 0x7fc75c13ee40 2024-03-29T11:23:45.376 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:45.397+0000 7fc743fff640 1 -- 172.21.15.84:0/3362748996 <== mon.1 v1:172.21.15.179:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4209477549) 0x7fc750001d40 con 0x7fc75c13ee40 2024-03-29T11:23:45.376 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:45.397+0000 7fc743fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:23:45.377 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:45.397+0000 7fc743fff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:23:45.377 INFO:teuthology.misc.health.smithi084.stdout: 2024-03-29T11:23:45.377 INFO:teuthology.misc.health.smithi084.stdout:{"epoch":10,"fsid":"2c3e002f-9eb0-4860-853b-4ef1ef562213","created":"2024-03-29T11:19:32.029169+0000","modified":"2024-03-29T11:19:34.200299+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:34.200299+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"5c39e276-2034-4716-bbc0-57a634eefa92","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"cd580e99-109f-4bf1-88c0-a5987bc8b9ed","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"11e60a5a-051d-453f-a190-66387cb35592","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"2c329fbe-bbc4-4030-bf87-d711cad04679","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f4213d4a-bd36-4433-91ae-d7e2b02481d3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"f0cf44ea-6055-48af-805b-aed858102ff9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"d895eaa0-cc0b-4ee4-9b5a-2dfcd89a60b7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"9bc5a7bc-8f52-4db9-96d6-de5e68d52eaa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:23:45.380 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:45.400+0000 7fc761a16640 10 librados: watch_flush enter 2024-03-29T11:23:45.380 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:45.400+0000 7fc761a16640 10 librados: watch_flush exit 2024-03-29T11:23:45.380 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:45.400+0000 7fc761a16640 20 client.4351.objecter shutdown clearing up homeless session... 2024-03-29T11:23:45.380 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:45.400+0000 7fc761a16640 10 client.4351.objecter successfully canceled tick 2024-03-29T11:23:45.380 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:45.400+0000 7fc761a16640 1 -- 172.21.15.84:0/3362748996 >> v1:172.21.15.179:6816/2981134980 conn(0x7fc74806dee0 legacy=0x7fc7480703a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:45.380 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:45.400+0000 7fc761a16640 10 monclient: shutdown 2024-03-29T11:23:45.380 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:45.400+0000 7fc761a16640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:23:45.380 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:45.400+0000 7fc761a16640 1 -- 172.21.15.84:0/3362748996 >> v1:172.21.15.179:6789/0 conn(0x7fc75c13ee40 legacy=0x7fc75c131bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:45.381 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:45.401+0000 7fc761a16640 1 -- 172.21.15.84:0/3362748996 shutdown_connections 2024-03-29T11:23:45.381 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:45.401+0000 7fc761a16640 1 -- 172.21.15.84:0/3362748996 >> 172.21.15.84:0/3362748996 conn(0x7fc75c06b660 msgr2=0x7fc75c0a9d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:23:45.384 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:45.404+0000 7fc761a16640 1 -- 172.21.15.84:0/3362748996 shutdown_connections 2024-03-29T11:23:45.384 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:45.404+0000 7fc761a16640 1 -- 172.21.15.84:0/3362748996 wait complete. 2024-03-29T11:23:45.384 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:45.404+0000 7fc761a16640 1 librados: shutdown 2024-03-29T11:23:45.403 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:23:52.210 DEBUG:teuthology.orchestra.run.smithi084:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:23:52.366 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:52.385+0000 7fdd0539a640 10 monclient: get_monmap_and_config 2024-03-29T11:23:52.366 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:52.385+0000 7fdd0539a640 10 monclient: build_initial_monmap 2024-03-29T11:23:52.366 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:52.385+0000 7fdd0539a640 10 monclient: monmap: 2024-03-29T11:23:52.366 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-29T11:23:52.367 INFO:teuthology.misc.health.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:23:52.367 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T11:23:52.386522+0000 2024-03-29T11:23:52.367 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T11:23:52.386522+0000 2024-03-29T11:23:52.367 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-29T11:23:52.367 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:23:52.367 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-29T11:23:52.367 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.noname-c 2024-03-29T11:23:52.367 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-b 2024-03-29T11:23:52.367 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:23:52.368 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:52.387+0000 7fdd0539a640 1 Processor -- start 2024-03-29T11:23:52.368 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:52.388+0000 7fdd0539a640 1 -- start start 2024-03-29T11:23:52.368 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:52.388+0000 7fdd0539a640 10 monclient: init 2024-03-29T11:23:52.370 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:52.389+0000 7fdd0539a640 10 monclient: _reopen_session rank -1 2024-03-29T11:23:52.370 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:52.389+0000 7fdd0539a640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:23:52.370 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:52.389+0000 7fdd0539a640 10 monclient(hunting): picked mon.noname-a con 0x7fdd000af0f0 addr v1:172.21.15.84:6789/0 2024-03-29T11:23:52.370 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:52.389+0000 7fdd0539a640 10 monclient(hunting): picked mon.noname-c con 0x7fdd000af4d0 addr v1:172.21.15.179:6789/0 2024-03-29T11:23:52.370 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:52.389+0000 7fdd0539a640 10 monclient(hunting): picked mon.noname-b con 0x7fdd000ab170 addr v1:172.21.15.84:6790/0 2024-03-29T11:23:52.370 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:52.389+0000 7fdd0539a640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdd0013f010 con 0x7fdd000af0f0 2024-03-29T11:23:52.370 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:52.389+0000 7fdd0539a640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdd000ab6c0 con 0x7fdd000af4d0 2024-03-29T11:23:52.371 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:52.389+0000 7fdd0539a640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdd000ab9a0 con 0x7fdd000ab170 2024-03-29T11:23:52.371 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:52.389+0000 7fdd0539a640 10 monclient(hunting): _renew_subs 2024-03-29T11:23:52.371 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:52.389+0000 7fdd0539a640 10 monclient(hunting): authenticate will time out at 1093.683834s 2024-03-29T11:23:52.374 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:52.392+0000 7fdcff7fe640 1 --1- >> v1:172.21.15.84:6790/0 conn(0x7fdd000ab170 0x7fdd0014a850 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.84:6790/0 says I am v1:172.21.15.84:37919/0 (socket says 172.21.15.84:37919) 2024-03-29T11:23:52.374 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:52.392+0000 7fdcff7fe640 1 -- 172.21.15.84:0/646468253 learned_addr learned my addr 172.21.15.84:0/646468253 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-29T11:23:52.374 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:52.394+0000 7fdcfdffb640 1 -- 172.21.15.84:0/646468253 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 681100445 0 0) 0x7fdd000ab9a0 con 0x7fdd000ab170 2024-03-29T11:23:52.374 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:52.394+0000 7fdcfdffb640 10 monclient(hunting): _negotiate 2024-03-29T11:23:52.375 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:52.394+0000 7fdcfdffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:23:52.375 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:52.394+0000 7fdcfdffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:23:52.375 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:52.394+0000 7fdcfdffb640 10 monclient(hunting): my global_id is 4391 2024-03-29T11:23:52.375 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:52.394+0000 7fdcfdffb640 1 -- 172.21.15.84:0/646468253 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdce8003520 con 0x7fdd000ab170 2024-03-29T11:23:52.375 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:52.395+0000 7fdcfdffb640 1 -- 172.21.15.84:0/646468253 <== mon.2 v1:172.21.15.84:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2200727928 0 0) 0x7fdce8003520 con 0x7fdd000ab170 2024-03-29T11:23:52.375 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:52.395+0000 7fdcfdffb640 1 -- 172.21.15.84:0/646468253 >> v1:172.21.15.179:6789/0 conn(0x7fdd000af4d0 legacy=0x7fdd0013f4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:52.376 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:52.395+0000 7fdcfdffb640 1 -- 172.21.15.84:0/646468253 >> v1:172.21.15.84:6789/0 conn(0x7fdd000af0f0 legacy=0x7fdd0013c3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:23:52.376 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:52.395+0000 7fdcfdffb640 10 monclient: _finish_hunting 0 2024-03-29T11:23:52.376 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:52.395+0000 7fdcfdffb640 1 monclient: found mon.noname-b 2024-03-29T11:23:52.376 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:52.395+0000 7fdcfdffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:23:52.376 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:52.395+0000 7fdcfdffb640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.84:6790/0 2024-03-29T11:23:52.376 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:52.395+0000 7fdcfdffb640 1 -- 172.21.15.84:0/646468253 --> v1:172.21.15.84:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd000abc80 con 0x7fdd000ab170 2024-03-29T11:23:52.376 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:52.395+0000 7fdcfdffb640 10 monclient: _finish_auth 0 2024-03-29T11:23:52.376 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:52.395+0000 7fdcfdffb640 10 monclient: _check_auth_tickets 2024-03-29T11:23:52.377 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:52.395+0000 7fdcfdffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:23:52.377 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:52.395+0000 7fdcfdffb640 1 -- 172.21.15.84:0/646468253 <== mon.2 v1:172.21.15.84:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4236868694 0 0) 0x7fdcf4001690 con 0x7fdd000ab170 2024-03-29T11:23:52.377 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:52.395+0000 7fdcfdffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:23:52.377 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:52.395+0000 7fdcfdffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:23:52.377 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:52.395+0000 7fdcfdffb640 10 monclient: dump: 2024-03-29T11:23:52.377 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T11:23:52.377 INFO:teuthology.misc.health.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:23:52.377 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T11:18:53.864025+0000 2024-03-29T11:23:52.378 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T11:18:53.864025+0000 2024-03-29T11:23:52.378 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T11:23:52.378 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:23:52.378 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T11:23:52.378 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T11:23:52.378 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T11:23:52.378 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:23:52.379 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:52.396+0000 7fdcfdffb640 1 -- 172.21.15.84:0/646468253 <== mon.2 v1:172.21.15.84:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fdcf4001920 con 0x7fdd000ab170 2024-03-29T11:23:52.379 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:52.396+0000 7fdcfdffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:23:52.380 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:52.396+0000 7fdcfdffb640 1 -- 172.21.15.84:0/646468253 <== mon.2 v1:172.21.15.84:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4236868694 0 0) 0x7fdcf4001cb0 con 0x7fdd000ab170 2024-03-29T11:23:52.380 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:52.396+0000 7fdcfdffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:23:52.380 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:52.396+0000 7fdcfdffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:23:52.380 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:52.396+0000 7fdcfdffb640 10 monclient: dump: 2024-03-29T11:23:52.380 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T11:23:52.380 INFO:teuthology.misc.health.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:23:52.380 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T11:18:53.864025+0000 2024-03-29T11:23:52.380 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T11:18:53.864025+0000 2024-03-29T11:23:52.380 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T11:23:52.380 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:23:52.380 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T11:23:52.380 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T11:23:52.380 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T11:23:52.380 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:23:52.380 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:52.397+0000 7fdd0539a640 5 monclient: authenticate success, global_id 4391 2024-03-29T11:23:52.381 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:52.397+0000 7fdd0539a640 10 monclient: get_monmap_and_config success 2024-03-29T11:23:52.381 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:52.397+0000 7fdd0539a640 4 set_mon_vals no callback set 2024-03-29T11:23:52.381 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:52.398+0000 7fdd0539a640 10 monclient: shutdown 2024-03-29T11:23:52.381 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:52.398+0000 7fdd0539a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:23:52.381 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:52.398+0000 7fdd0539a640 1 -- 172.21.15.84:0/646468253 >> v1:172.21.15.84:6790/0 conn(0x7fdd000ab170 legacy=0x7fdd0014a850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:52.381 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:52.399+0000 7fdd0539a640 1 -- 172.21.15.84:0/646468253 shutdown_connections 2024-03-29T11:23:52.381 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:52.399+0000 7fdd0539a640 1 -- 172.21.15.84:0/646468253 >> 172.21.15.84:0/646468253 conn(0x7fdd000b35e0 msgr2=0x7fdd000b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:23:52.381 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:52.399+0000 7fdd0539a640 1 -- 172.21.15.84:0/646468253 shutdown_connections 2024-03-29T11:23:52.381 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:52.399+0000 7fdd0539a640 1 -- 172.21.15.84:0/646468253 wait complete. 2024-03-29T11:23:52.382 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:52.400+0000 7fdd0539a640 10 monclient: build_initial_monmap 2024-03-29T11:23:52.382 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:52.400+0000 7fdd0539a640 10 monclient: monmap: 2024-03-29T11:23:52.382 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-29T11:23:52.382 INFO:teuthology.misc.health.smithi084.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:23:52.382 INFO:teuthology.misc.health.smithi084.stderr:last_changed 0.000000 2024-03-29T11:23:52.382 INFO:teuthology.misc.health.smithi084.stderr:created 0.000000 2024-03-29T11:23:52.382 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-29T11:23:52.383 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:23:52.383 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-29T11:23:52.383 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.noname-b 2024-03-29T11:23:52.383 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-c 2024-03-29T11:23:52.383 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:23:52.383 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:52.401+0000 7fdd0539a640 1 librados: starting msgr at 2024-03-29T11:23:52.383 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:52.401+0000 7fdd0539a640 1 librados: starting objecter 2024-03-29T11:23:52.383 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:52.401+0000 7fdd0539a640 1 Processor -- start 2024-03-29T11:23:52.383 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:52.401+0000 7fdd0539a640 1 -- start start 2024-03-29T11:23:52.383 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:52.401+0000 7fdd0539a640 1 librados: setting wanted keys 2024-03-29T11:23:52.383 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:52.401+0000 7fdd0539a640 1 librados: calling monclient init 2024-03-29T11:23:52.383 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:52.401+0000 7fdd0539a640 10 monclient: init 2024-03-29T11:23:52.383 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:52.402+0000 7fdd0539a640 10 monclient: _reopen_session rank -1 2024-03-29T11:23:52.383 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:52.402+0000 7fdd0539a640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:23:52.385 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:52.405+0000 7fdd0539a640 10 monclient(hunting): picked mon.noname-c con 0x7fdd000ab170 addr v1:172.21.15.84:6790/0 2024-03-29T11:23:52.389 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:52.408+0000 7fdd0539a640 10 monclient(hunting): picked mon.noname-b con 0x7fdd000af0f0 addr v1:172.21.15.179:6789/0 2024-03-29T11:23:52.389 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:52.408+0000 7fdd0539a640 10 monclient(hunting): picked mon.noname-a con 0x7fdd001e1820 addr v1:172.21.15.84:6789/0 2024-03-29T11:23:52.389 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:52.408+0000 7fdd0539a640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdd0013bb20 con 0x7fdd001e1820 2024-03-29T11:23:52.389 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:52.408+0000 7fdd0539a640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdd0013bcb0 con 0x7fdd000af0f0 2024-03-29T11:23:52.389 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:52.408+0000 7fdd0539a640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdd0013e8a0 con 0x7fdd000ab170 2024-03-29T11:23:52.390 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:52.408+0000 7fdd0539a640 10 monclient(hunting): _renew_subs 2024-03-29T11:23:52.390 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:52.408+0000 7fdd0539a640 10 monclient(hunting): authenticate will time out at 1093.702663s 2024-03-29T11:23:52.390 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:52.408+0000 7fdcfeffd640 1 --1- >> v1:172.21.15.84:6790/0 conn(0x7fdd000ab170 0x7fdd001e7420 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.84:6790/0 says I am v1:172.21.15.84:38023/0 (socket says 172.21.15.84:38023) 2024-03-29T11:23:52.390 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:52.408+0000 7fdcfeffd640 1 -- 172.21.15.84:0/3794045161 learned_addr learned my addr 172.21.15.84:0/3794045161 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-29T11:23:52.390 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:52.409+0000 7fdce77fe640 10 client.?.objecter ms_handle_connect 0x7fdd000ab170 2024-03-29T11:23:52.390 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:52.409+0000 7fdce77fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:23:52.393 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:52.412+0000 7fdce77fe640 10 client.?.objecter ms_handle_connect 0x7fdd000af0f0 2024-03-29T11:23:52.393 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:52.412+0000 7fdce77fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:23:52.393 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:52.412+0000 7fdce77fe640 1 -- 172.21.15.84:0/3794045161 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3112020188 0 0) 0x7fdd0013e8a0 con 0x7fdd000ab170 2024-03-29T11:23:52.394 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:52.412+0000 7fdce77fe640 10 monclient(hunting): _negotiate 2024-03-29T11:23:52.394 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:52.412+0000 7fdce77fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:23:52.394 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:52.412+0000 7fdce77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:23:52.394 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:52.412+0000 7fdce77fe640 10 monclient(hunting): my global_id is 4394 2024-03-29T11:23:52.394 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:52.412+0000 7fdce77fe640 1 -- 172.21.15.84:0/3794045161 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdcec004eb0 con 0x7fdd000ab170 2024-03-29T11:23:52.394 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:52.413+0000 7fdce77fe640 1 -- 172.21.15.84:0/3794045161 <== mon.2 v1:172.21.15.84:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4067816381 0 0) 0x7fdcec004eb0 con 0x7fdd000ab170 2024-03-29T11:23:52.394 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:52.413+0000 7fdce77fe640 1 -- 172.21.15.84:0/3794045161 --> v1:172.21.15.84:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fdd0013e8a0 con 0x7fdd000ab170 2024-03-29T11:23:52.394 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:52.413+0000 7fdce77fe640 1 -- 172.21.15.84:0/3794045161 <== mon.2 v1:172.21.15.84:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4236868694 0 0) 0x7fdcf0004080 con 0x7fdd000ab170 2024-03-29T11:23:52.394 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:52.413+0000 7fdce77fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:23:52.394 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:52.413+0000 7fdce77fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:23:52.394 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:52.413+0000 7fdce77fe640 10 monclient(hunting): dump: 2024-03-29T11:23:52.394 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T11:23:52.395 INFO:teuthology.misc.health.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:23:52.395 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T11:18:53.864025+0000 2024-03-29T11:23:52.395 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T11:18:53.864025+0000 2024-03-29T11:23:52.395 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T11:23:52.395 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:23:52.395 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T11:23:52.395 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T11:23:52.395 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T11:23:52.395 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:23:52.395 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:52.413+0000 7fdce77fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:23:52.395 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:52.413+0000 7fdce77fe640 10 client.?.objecter ms_handle_connect 0x7fdd001e1820 2024-03-29T11:23:52.395 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:52.413+0000 7fdce77fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:23:52.395 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:52.413+0000 7fdce77fe640 1 -- 172.21.15.84:0/3794045161 <== mon.0 v1:172.21.15.84:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2849058123 0 0) 0x7fdd0013bb20 con 0x7fdd001e1820 2024-03-29T11:23:52.395 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:52.414+0000 7fdce77fe640 10 monclient(hunting): _negotiate 2024-03-29T11:23:52.395 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:52.414+0000 7fdce77fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:23:52.396 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:52.414+0000 7fdce77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:23:52.396 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:52.414+0000 7fdce77fe640 10 monclient(hunting): my global_id is 4401 2024-03-29T11:23:52.396 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:52.414+0000 7fdce77fe640 1 -- 172.21.15.84:0/3794045161 --> v1:172.21.15.84:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdcec005680 con 0x7fdd001e1820 2024-03-29T11:23:52.396 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:52.414+0000 7fdce77fe640 1 -- 172.21.15.84:0/3794045161 <== mon.0 v1:172.21.15.84:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3602871664 0 0) 0x7fdcec005680 con 0x7fdd001e1820 2024-03-29T11:23:52.397 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:52.414+0000 7fdce77fe640 1 -- 172.21.15.84:0/3794045161 --> v1:172.21.15.84:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fdcec005b10 con 0x7fdd001e1820 2024-03-29T11:23:52.397 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:52.414+0000 7fdce77fe640 1 -- 172.21.15.84:0/3794045161 <== mon.0 v1:172.21.15.84:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4236868694 0 0) 0x7fdcf40013b0 con 0x7fdd001e1820 2024-03-29T11:23:52.397 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:52.414+0000 7fdce77fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:23:52.397 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:52.414+0000 7fdce77fe640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:23:52.397 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:52.414+0000 7fdce77fe640 10 monclient(hunting): dump: 2024-03-29T11:23:52.397 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T11:23:52.397 INFO:teuthology.misc.health.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:23:52.397 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T11:18:53.864025+0000 2024-03-29T11:23:52.397 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T11:18:53.864025+0000 2024-03-29T11:23:52.397 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T11:23:52.397 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:23:52.397 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T11:23:52.397 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T11:23:52.398 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T11:23:52.398 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:23:52.398 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:52.414+0000 7fdce77fe640 1 -- 172.21.15.84:0/3794045161 <== mon.0 v1:172.21.15.84:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3673758821 0 0) 0x7fdcec005b10 con 0x7fdd001e1820 2024-03-29T11:23:52.398 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:52.414+0000 7fdce77fe640 1 -- 172.21.15.84:0/3794045161 >> v1:172.21.15.84:6790/0 conn(0x7fdd000ab170 legacy=0x7fdd001e7420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:52.398 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:52.414+0000 7fdce77fe640 1 -- 172.21.15.84:0/3794045161 >> v1:172.21.15.179:6789/0 conn(0x7fdd000af0f0 legacy=0x7fdd001e1110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:52.398 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:52.414+0000 7fdce77fe640 10 monclient: _finish_hunting 0 2024-03-29T11:23:52.398 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:52.414+0000 7fdce77fe640 1 monclient: found mon.a 2024-03-29T11:23:52.398 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:52.414+0000 7fdce77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:23:52.398 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:52.415+0000 7fdce77fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.84:6789/0 2024-03-29T11:23:52.398 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:52.415+0000 7fdce77fe640 1 -- 172.21.15.84:0/3794045161 --> v1:172.21.15.84:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd0013ea30 con 0x7fdd001e1820 2024-03-29T11:23:52.398 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:52.415+0000 7fdce77fe640 10 monclient: _finish_auth 0 2024-03-29T11:23:52.398 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:52.415+0000 7fdce77fe640 10 monclient: _check_auth_tickets 2024-03-29T11:23:52.398 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:52.415+0000 7fdce77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:23:52.399 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:52.416+0000 7fdd0539a640 5 monclient: authenticate success, global_id 4401 2024-03-29T11:23:52.399 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:52.416+0000 7fdd0539a640 10 monclient: _renew_subs 2024-03-29T11:23:52.399 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:52.416+0000 7fdd0539a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.84:6789/0 2024-03-29T11:23:52.399 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:52.416+0000 7fdd0539a640 1 -- 172.21.15.84:0/3794045161 --> v1:172.21.15.84:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdd0006e7a0 con 0x7fdd001e1820 2024-03-29T11:23:52.399 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:52.416+0000 7fdd0539a640 10 client.4401.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:23:52.399 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:52.416+0000 7fdd0539a640 10 monclient: _renew_subs 2024-03-29T11:23:52.399 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:52.416+0000 7fdd0539a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.84:6789/0 2024-03-29T11:23:52.399 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:52.416+0000 7fdd0539a640 1 -- 172.21.15.84:0/3794045161 --> v1:172.21.15.84:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fdd0006f4d0 con 0x7fdd001e1820 2024-03-29T11:23:52.399 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:52.416+0000 7fdd0539a640 1 librados: init done 2024-03-29T11:23:52.400 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:52.418+0000 7fdce77fe640 1 -- 172.21.15.84:0/3794045161 <== mon.0 v1:172.21.15.84:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fdcf4001210 con 0x7fdd001e1820 2024-03-29T11:23:52.400 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:52.418+0000 7fdce77fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:23:52.400 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:52.418+0000 7fdce77fe640 1 -- 172.21.15.84:0/3794045161 <== mon.0 v1:172.21.15.84:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4236868694 0 0) 0x7fdcf40013b0 con 0x7fdd001e1820 2024-03-29T11:23:52.401 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:52.418+0000 7fdce77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:23:52.401 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:52.418+0000 7fdce77fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:23:52.401 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:52.418+0000 7fdce77fe640 10 monclient: dump: 2024-03-29T11:23:52.401 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T11:23:52.401 INFO:teuthology.misc.health.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:23:52.401 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T11:18:53.864025+0000 2024-03-29T11:23:52.401 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T11:18:53.864025+0000 2024-03-29T11:23:52.401 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T11:23:52.401 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:23:52.401 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T11:23:52.402 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T11:23:52.402 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T11:23:52.402 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:23:52.402 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:52.418+0000 7fdcfd7fa640 4 set_mon_vals no callback set 2024-03-29T11:23:52.403 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:52.419+0000 7fdce77fe640 1 -- 172.21.15.84:0/3794045161 <== mon.0 v1:172.21.15.84:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4033942909 0 0) 0x7fdcf401a7b0 con 0x7fdd001e1820 2024-03-29T11:23:52.403 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:52.420+0000 7fdd0539a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:23:52.403 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:52.420+0000 7fdd0539a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:23:52.403 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:52.420+0000 7fdd0539a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.84:6789/0 2024-03-29T11:23:52.404 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:52.420+0000 7fdd0539a640 1 -- 172.21.15.84:0/3794045161 --> v1:172.21.15.84:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdce0005080 con 0x7fdd001e1820 2024-03-29T11:23:52.404 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:52.420+0000 7fdcfeffd640 10 monclient: get_auth_request con 0x7fdcec06e640 auth_method 0 2024-03-29T11:23:52.404 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:52.420+0000 7fdce77fe640 1 -- 172.21.15.84:0/3794045161 <== mon.0 v1:172.21.15.84:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2808139587 0 0) 0x7fdcf40542c0 con 0x7fdd001e1820 2024-03-29T11:23:52.404 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:52.420+0000 7fdce77fe640 10 client.4401.objecter ms_dispatch 0x7fdd00000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:23:52.404 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:52.420+0000 7fdce77fe640 3 client.4401.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:23:52.404 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:52.420+0000 7fdce77fe640 3 client.4401.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:23:52.404 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:52.420+0000 7fdce77fe640 20 client.4401.objecter dump_active .. 0 homeless 2024-03-29T11:23:52.404 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:52.421+0000 7fdce77fe640 10 client.4401.objecter ms_handle_connect 0x7fdcec06e640 2024-03-29T11:23:52.412 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:52.432+0000 7fdce77fe640 1 -- 172.21.15.84:0/3794045161 <== mon.0 v1:172.21.15.84:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fdcf4050560 con 0x7fdd001e1820 2024-03-29T11:23:52.412 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:52.432+0000 7fdce77fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:23:52.412 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:52.432+0000 7fdce77fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:23:52.747 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:52.766+0000 7fdd0539a640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:23:52.747 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:52.766+0000 7fdd0539a640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:23:52.747 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:52.766+0000 7fdd0539a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.84:6789/0 2024-03-29T11:23:52.748 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:52.766+0000 7fdd0539a640 1 -- 172.21.15.84:0/3794045161 --> v1:172.21.15.84:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fdce0005330 con 0x7fdd001e1820 2024-03-29T11:23:52.748 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:52.767+0000 7fdce77fe640 1 -- 172.21.15.84:0/3794045161 <== mon.0 v1:172.21.15.84:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4209477549) 0x7fdcf4050700 con 0x7fdd001e1820 2024-03-29T11:23:52.748 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:52.767+0000 7fdce77fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:23:52.748 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:52.767+0000 7fdce77fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:23:52.748 INFO:teuthology.misc.health.smithi084.stdout: 2024-03-29T11:23:52.749 INFO:teuthology.misc.health.smithi084.stdout:{"epoch":10,"fsid":"2c3e002f-9eb0-4860-853b-4ef1ef562213","created":"2024-03-29T11:19:32.029169+0000","modified":"2024-03-29T11:19:34.200299+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:34.200299+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"5c39e276-2034-4716-bbc0-57a634eefa92","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"cd580e99-109f-4bf1-88c0-a5987bc8b9ed","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"11e60a5a-051d-453f-a190-66387cb35592","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"2c329fbe-bbc4-4030-bf87-d711cad04679","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f4213d4a-bd36-4433-91ae-d7e2b02481d3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"f0cf44ea-6055-48af-805b-aed858102ff9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"d895eaa0-cc0b-4ee4-9b5a-2dfcd89a60b7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"9bc5a7bc-8f52-4db9-96d6-de5e68d52eaa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:23:52.752 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:52.771+0000 7fdd0539a640 10 librados: watch_flush enter 2024-03-29T11:23:52.752 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:52.771+0000 7fdd0539a640 10 librados: watch_flush exit 2024-03-29T11:23:52.752 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:52.771+0000 7fdd0539a640 20 client.4401.objecter shutdown clearing up homeless session... 2024-03-29T11:23:52.752 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:52.771+0000 7fdd0539a640 10 client.4401.objecter successfully canceled tick 2024-03-29T11:23:52.752 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:52.771+0000 7fdd0539a640 1 -- 172.21.15.84:0/3794045161 >> v1:172.21.15.179:6816/2981134980 conn(0x7fdcec06e640 legacy=0x7fdcec070b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:52.752 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:52.771+0000 7fdd0539a640 10 monclient: shutdown 2024-03-29T11:23:52.752 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:52.771+0000 7fdd0539a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:23:52.752 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:52.771+0000 7fdd0539a640 1 -- 172.21.15.84:0/3794045161 >> v1:172.21.15.84:6789/0 conn(0x7fdd001e1820 legacy=0x7fdd001ef760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:52.753 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:52.772+0000 7fdd0539a640 1 -- 172.21.15.84:0/3794045161 shutdown_connections 2024-03-29T11:23:52.753 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:52.772+0000 7fdd0539a640 1 -- 172.21.15.84:0/3794045161 >> 172.21.15.84:0/3794045161 conn(0x7fdd000b35e0 msgr2=0x7fdd001ccf60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:23:52.753 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:52.772+0000 7fdd0539a640 1 -- 172.21.15.84:0/3794045161 shutdown_connections 2024-03-29T11:23:52.753 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:52.772+0000 7fdd0539a640 1 -- 172.21.15.84:0/3794045161 wait complete. 2024-03-29T11:23:52.753 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:52.773+0000 7fdd0539a640 1 librados: shutdown 2024-03-29T11:23:52.770 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:23:59.576 DEBUG:teuthology.orchestra.run.smithi084:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:23:59.721 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:59.741+0000 7f2207a0f640 10 monclient: get_monmap_and_config 2024-03-29T11:23:59.721 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:59.741+0000 7f2207a0f640 10 monclient: build_initial_monmap 2024-03-29T11:23:59.721 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:59.741+0000 7f2207a0f640 10 monclient: monmap: 2024-03-29T11:23:59.721 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-29T11:23:59.721 INFO:teuthology.misc.health.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:23:59.721 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T11:23:59.742019+0000 2024-03-29T11:23:59.722 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T11:23:59.742019+0000 2024-03-29T11:23:59.722 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-29T11:23:59.722 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:23:59.722 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-29T11:23:59.722 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.noname-c 2024-03-29T11:23:59.722 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-b 2024-03-29T11:23:59.722 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:23:59.723 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:59.742+0000 7f2207a0f640 1 Processor -- start 2024-03-29T11:23:59.723 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:59.742+0000 7f2207a0f640 1 -- start start 2024-03-29T11:23:59.723 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:59.742+0000 7f2207a0f640 10 monclient: init 2024-03-29T11:23:59.724 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:59.743+0000 7f2207a0f640 10 monclient: _reopen_session rank -1 2024-03-29T11:23:59.724 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:59.743+0000 7f2207a0f640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:23:59.724 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:59.743+0000 7f2207a0f640 10 monclient(hunting): picked mon.noname-c con 0x7f22001422b0 addr v1:172.21.15.179:6789/0 2024-03-29T11:23:59.725 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:59.743+0000 7f2207a0f640 10 monclient(hunting): picked mon.noname-b con 0x7f2200142690 addr v1:172.21.15.84:6790/0 2024-03-29T11:23:59.725 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:59.743+0000 7f2207a0f640 10 monclient(hunting): picked mon.noname-a con 0x7f22000b17d0 addr v1:172.21.15.84:6789/0 2024-03-29T11:23:59.725 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:59.743+0000 7f2207a0f640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2200142c80 con 0x7f22000b17d0 2024-03-29T11:23:59.725 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:59.743+0000 7f2207a0f640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f22000aae40 con 0x7f22001422b0 2024-03-29T11:23:59.725 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:59.743+0000 7f2205fac640 1 --1- >> v1:172.21.15.84:6789/0 conn(0x7f22000b17d0 0x7f22000b2ca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.84:6789/0 says I am v1:172.21.15.84:57976/0 (socket says 172.21.15.84:57976) 2024-03-29T11:23:59.725 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:59.744+0000 7f2205fac640 1 -- 172.21.15.84:0/3797538811 learned_addr learned my addr 172.21.15.84:0/3797538811 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-29T11:23:59.725 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:59.744+0000 7f2207a0f640 1 -- 172.21.15.84:0/3797538811 --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f220013fbd0 con 0x7f2200142690 2024-03-29T11:23:59.725 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:59.744+0000 7f2207a0f640 10 monclient(hunting): _renew_subs 2024-03-29T11:23:59.725 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:59.744+0000 7f2207a0f640 10 monclient(hunting): authenticate will time out at 1101.038440s 2024-03-29T11:23:59.725 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:59.744+0000 7f21f7fff640 1 -- 172.21.15.84:0/3797538811 <== mon.0 v1:172.21.15.84:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1451858272 0 0) 0x7f2200142c80 con 0x7f22000b17d0 2024-03-29T11:23:59.725 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:59.744+0000 7f21f7fff640 10 monclient(hunting): _negotiate 2024-03-29T11:23:59.725 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:59.744+0000 7f21f7fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:23:59.725 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:59.744+0000 7f21f7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:23:59.725 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:59.744+0000 7f21f7fff640 10 monclient(hunting): my global_id is 4404 2024-03-29T11:23:59.725 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:59.744+0000 7f21f7fff640 1 -- 172.21.15.84:0/3797538811 --> v1:172.21.15.84:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f21ec003520 con 0x7f22000b17d0 2024-03-29T11:23:59.726 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:59.744+0000 7f21f7fff640 1 -- 172.21.15.84:0/3797538811 <== mon.1 v1:172.21.15.179:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 187669897 0 0) 0x7f22000aae40 con 0x7f22001422b0 2024-03-29T11:23:59.726 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:59.744+0000 7f21f7fff640 10 monclient(hunting): _negotiate 2024-03-29T11:23:59.726 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:59.744+0000 7f21f7fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:23:59.726 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:59.744+0000 7f21f7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:23:59.726 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:59.744+0000 7f21f7fff640 10 monclient(hunting): my global_id is 4357 2024-03-29T11:23:59.726 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:59.744+0000 7f21f7fff640 1 -- 172.21.15.84:0/3797538811 --> v1:172.21.15.179:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2200142c80 con 0x7f22001422b0 2024-03-29T11:23:59.727 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:59.745+0000 7f21f7fff640 1 -- 172.21.15.84:0/3797538811 <== mon.0 v1:172.21.15.84:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3241078321 0 0) 0x7f21ec003520 con 0x7f22000b17d0 2024-03-29T11:23:59.727 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:59.745+0000 7f21f7fff640 1 -- 172.21.15.84:0/3797538811 >> v1:172.21.15.84:6790/0 conn(0x7f2200142690 legacy=0x7f22000b2590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:23:59.727 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:59.745+0000 7f21f7fff640 1 -- 172.21.15.84:0/3797538811 >> v1:172.21.15.179:6789/0 conn(0x7f22001422b0 legacy=0x7f220013f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:59.727 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:59.745+0000 7f21f7fff640 10 monclient: _finish_hunting 0 2024-03-29T11:23:59.727 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:59.745+0000 7f21f7fff640 1 monclient: found mon.noname-a 2024-03-29T11:23:59.727 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:59.745+0000 7f21f7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:23:59.727 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:59.745+0000 7f21f7fff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.84:6789/0 2024-03-29T11:23:59.727 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:59.745+0000 7f21f7fff640 1 -- 172.21.15.84:0/3797538811 --> v1:172.21.15.84:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2200151940 con 0x7f22000b17d0 2024-03-29T11:23:59.727 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:59.745+0000 7f21f7fff640 10 monclient: _finish_auth 0 2024-03-29T11:23:59.727 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:59.745+0000 7f21f7fff640 10 monclient: _check_auth_tickets 2024-03-29T11:23:59.727 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:59.745+0000 7f21f7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:23:59.727 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:59.745+0000 7f21f7fff640 1 -- 172.21.15.84:0/3797538811 <== mon.0 v1:172.21.15.84:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4236868694 0 0) 0x7f21fc001690 con 0x7f22000b17d0 2024-03-29T11:23:59.727 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:59.745+0000 7f21f7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:23:59.727 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:59.745+0000 7f21f7fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:23:59.727 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:59.745+0000 7f21f7fff640 10 monclient: dump: 2024-03-29T11:23:59.728 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T11:23:59.728 INFO:teuthology.misc.health.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:23:59.728 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T11:18:53.864025+0000 2024-03-29T11:23:59.728 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T11:18:53.864025+0000 2024-03-29T11:23:59.728 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T11:23:59.728 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:23:59.728 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T11:23:59.728 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T11:23:59.728 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T11:23:59.728 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:23:59.728 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:59.745+0000 7f2207a0f640 5 monclient: authenticate success, global_id 4404 2024-03-29T11:23:59.728 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:59.745+0000 7f2207a0f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:23:59.728 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:59.745+0000 7f21f7fff640 1 -- 172.21.15.84:0/3797538811 <== mon.0 v1:172.21.15.84:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f21fc0017f0 con 0x7f22000b17d0 2024-03-29T11:23:59.728 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:59.745+0000 7f21f7fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:23:59.729 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:59.745+0000 7f21f7fff640 1 -- 172.21.15.84:0/3797538811 <== mon.0 v1:172.21.15.84:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4236868694 0 0) 0x7f21fc001b80 con 0x7f22000b17d0 2024-03-29T11:23:59.729 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:59.745+0000 7f21f7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:23:59.729 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:59.745+0000 7f21f7fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:23:59.729 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:59.745+0000 7f21f7fff640 10 monclient: dump: 2024-03-29T11:23:59.729 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T11:23:59.729 INFO:teuthology.misc.health.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:23:59.729 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T11:18:53.864025+0000 2024-03-29T11:23:59.729 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T11:18:53.864025+0000 2024-03-29T11:23:59.729 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T11:23:59.729 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:23:59.729 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T11:23:59.729 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T11:23:59.729 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T11:23:59.729 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:23:59.730 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:59.745+0000 7f2207a0f640 10 monclient: get_monmap_and_config success 2024-03-29T11:23:59.730 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:59.745+0000 7f2207a0f640 4 set_mon_vals no callback set 2024-03-29T11:23:59.736 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:59.746+0000 7f2207a0f640 10 monclient: shutdown 2024-03-29T11:23:59.736 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:59.746+0000 7f2207a0f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:23:59.736 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:59.746+0000 7f2207a0f640 1 -- 172.21.15.84:0/3797538811 >> v1:172.21.15.84:6789/0 conn(0x7f22000b17d0 legacy=0x7f22000b2ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:59.737 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:59.746+0000 7f2207a0f640 1 -- 172.21.15.84:0/3797538811 shutdown_connections 2024-03-29T11:23:59.737 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:59.746+0000 7f2207a0f640 1 -- 172.21.15.84:0/3797538811 >> 172.21.15.84:0/3797538811 conn(0x7f2200066a00 msgr2=0x7f2200066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:23:59.737 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:59.747+0000 7f2207a0f640 1 -- 172.21.15.84:0/3797538811 shutdown_connections 2024-03-29T11:23:59.737 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:59.747+0000 7f2207a0f640 1 -- 172.21.15.84:0/3797538811 wait complete. 2024-03-29T11:23:59.737 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:59.748+0000 7f2207a0f640 10 monclient: build_initial_monmap 2024-03-29T11:23:59.737 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:59.748+0000 7f2207a0f640 10 monclient: monmap: 2024-03-29T11:23:59.737 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-29T11:23:59.737 INFO:teuthology.misc.health.smithi084.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:23:59.737 INFO:teuthology.misc.health.smithi084.stderr:last_changed 0.000000 2024-03-29T11:23:59.737 INFO:teuthology.misc.health.smithi084.stderr:created 0.000000 2024-03-29T11:23:59.737 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-29T11:23:59.737 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:23:59.737 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-29T11:23:59.737 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.noname-b 2024-03-29T11:23:59.737 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-c 2024-03-29T11:23:59.738 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:23:59.738 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:59.748+0000 7f2207a0f640 1 librados: starting msgr at 2024-03-29T11:23:59.738 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:59.748+0000 7f2207a0f640 1 librados: starting objecter 2024-03-29T11:23:59.738 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:59.748+0000 7f2207a0f640 1 Processor -- start 2024-03-29T11:23:59.738 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:59.749+0000 7f2207a0f640 1 -- start start 2024-03-29T11:23:59.738 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:59.749+0000 7f2207a0f640 1 librados: setting wanted keys 2024-03-29T11:23:59.738 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:59.749+0000 7f2207a0f640 1 librados: calling monclient init 2024-03-29T11:23:59.738 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:59.749+0000 7f2207a0f640 10 monclient: init 2024-03-29T11:23:59.738 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:59.749+0000 7f2207a0f640 10 monclient: _reopen_session rank -1 2024-03-29T11:23:59.738 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:59.749+0000 7f2207a0f640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:23:59.738 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:59.749+0000 7f2207a0f640 10 monclient(hunting): picked mon.noname-c con 0x7f22000b17d0 addr v1:172.21.15.84:6790/0 2024-03-29T11:23:59.738 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:59.750+0000 7f2207a0f640 10 monclient(hunting): picked mon.noname-a con 0x7f22001422b0 addr v1:172.21.15.84:6789/0 2024-03-29T11:23:59.738 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:59.750+0000 7f2207a0f640 10 monclient(hunting): picked mon.noname-b con 0x7f2200142690 addr v1:172.21.15.179:6789/0 2024-03-29T11:23:59.738 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:59.750+0000 7f2207a0f640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f22000b0b40 con 0x7f22001422b0 2024-03-29T11:23:59.739 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:59.750+0000 7f2205fac640 1 --1- >> v1:172.21.15.179:6789/0 conn(0x7f2200142690 0x7f22000acf60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.179:6789/0 says I am v1:172.21.15.84:47378/0 (socket says 172.21.15.84:47378) 2024-03-29T11:23:59.739 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:59.750+0000 7f2205fac640 1 -- 172.21.15.84:0/1679283077 learned_addr learned my addr 172.21.15.84:0/1679283077 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-29T11:23:59.739 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:59.750+0000 7f2207a0f640 1 -- 172.21.15.84:0/1679283077 --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f22000b0d90 con 0x7f2200142690 2024-03-29T11:23:59.739 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:59.750+0000 7f21f5ffb640 10 client.?.objecter ms_handle_connect 0x7f2200142690 2024-03-29T11:23:59.739 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:59.750+0000 7f21f5ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:23:59.739 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:59.750+0000 7f2207a0f640 1 -- 172.21.15.84:0/1679283077 --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f22000b1070 con 0x7f22000b17d0 2024-03-29T11:23:59.739 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:59.750+0000 7f2207a0f640 10 monclient(hunting): _renew_subs 2024-03-29T11:23:59.739 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:59.750+0000 7f2207a0f640 10 monclient(hunting): authenticate will time out at 1101.044824s 2024-03-29T11:23:59.739 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:59.750+0000 7f21f5ffb640 10 client.?.objecter ms_handle_connect 0x7f22001422b0 2024-03-29T11:23:59.739 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:59.751+0000 7f21f5ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:23:59.739 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:59.751+0000 7f21f5ffb640 1 -- 172.21.15.84:0/1679283077 <== mon.1 v1:172.21.15.179:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3087230492 0 0) 0x7f22000b0d90 con 0x7f2200142690 2024-03-29T11:23:59.739 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:59.751+0000 7f21f5ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:23:59.739 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:59.751+0000 7f21f5ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:23:59.739 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:59.751+0000 7f21f5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:23:59.740 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:59.751+0000 7f21f5ffb640 10 monclient(hunting): my global_id is 4360 2024-03-29T11:23:59.740 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:59.751+0000 7f21f5ffb640 1 -- 172.21.15.84:0/1679283077 --> v1:172.21.15.179:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f21e4004eb0 con 0x7f2200142690 2024-03-29T11:23:59.740 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:59.751+0000 7f21f5ffb640 1 -- 172.21.15.84:0/1679283077 <== mon.1 v1:172.21.15.179:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 605273162 0 0) 0x7f21e4004eb0 con 0x7f2200142690 2024-03-29T11:23:59.740 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:59.751+0000 7f21f5ffb640 1 -- 172.21.15.84:0/1679283077 --> v1:172.21.15.179:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f22000b0d90 con 0x7f2200142690 2024-03-29T11:23:59.740 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:59.751+0000 7f21f5ffb640 1 -- 172.21.15.84:0/1679283077 <== mon.1 v1:172.21.15.179:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4236868694 0 0) 0x7f21fc001b00 con 0x7f2200142690 2024-03-29T11:23:59.740 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:59.751+0000 7f21f5ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:23:59.740 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:59.751+0000 7f21f5ffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:23:59.740 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:59.751+0000 7f21f5ffb640 10 monclient(hunting): dump: 2024-03-29T11:23:59.740 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T11:23:59.740 INFO:teuthology.misc.health.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:23:59.740 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T11:18:53.864025+0000 2024-03-29T11:23:59.740 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T11:18:53.864025+0000 2024-03-29T11:23:59.740 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T11:23:59.740 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:23:59.740 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T11:23:59.740 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T11:23:59.741 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T11:23:59.741 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:23:59.741 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:59.751+0000 7f21f5ffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:23:59.741 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:59.751+0000 7f21f5ffb640 10 client.?.objecter ms_handle_connect 0x7f22000b17d0 2024-03-29T11:23:59.741 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:59.751+0000 7f21f5ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:23:59.741 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:59.751+0000 7f21f5ffb640 1 -- 172.21.15.84:0/1679283077 <== mon.1 v1:172.21.15.179:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1335860411 0 0) 0x7f22000b0d90 con 0x7f2200142690 2024-03-29T11:23:59.741 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:59.751+0000 7f21f5ffb640 1 -- 172.21.15.84:0/1679283077 >> v1:172.21.15.84:6790/0 conn(0x7f22000b17d0 legacy=0x7f2200136ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:59.741 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:59.751+0000 7f21f5ffb640 1 -- 172.21.15.84:0/1679283077 >> v1:172.21.15.84:6789/0 conn(0x7f22001422b0 legacy=0x7f22000ac850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:59.741 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:59.751+0000 7f21f5ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:23:59.741 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:59.751+0000 7f21f5ffb640 1 monclient: found mon.b 2024-03-29T11:23:59.741 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:59.751+0000 7f21f5ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:23:59.741 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:59.751+0000 7f21f5ffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-29T11:23:59.741 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:59.751+0000 7f21f5ffb640 1 -- 172.21.15.84:0/1679283077 --> v1:172.21.15.179:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22000aef60 con 0x7f2200142690 2024-03-29T11:23:59.741 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:59.751+0000 7f21f5ffb640 10 monclient: _finish_auth 0 2024-03-29T11:23:59.742 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:59.751+0000 7f21f5ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:23:59.742 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:59.751+0000 7f21f5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:23:59.742 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:59.751+0000 7f2207a0f640 5 monclient: authenticate success, global_id 4360 2024-03-29T11:23:59.742 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:59.751+0000 7f2207a0f640 10 monclient: _renew_subs 2024-03-29T11:23:59.742 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:59.751+0000 7f2207a0f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-29T11:23:59.742 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:59.751+0000 7f2207a0f640 1 -- 172.21.15.84:0/1679283077 --> v1:172.21.15.179:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f22000af1d0 con 0x7f2200142690 2024-03-29T11:23:59.742 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:59.752+0000 7f2207a0f640 10 client.4360.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:23:59.742 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:59.752+0000 7f2207a0f640 10 monclient: _renew_subs 2024-03-29T11:23:59.742 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:59.752+0000 7f2207a0f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-29T11:23:59.742 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:59.752+0000 7f2207a0f640 1 -- 172.21.15.84:0/1679283077 --> v1:172.21.15.179:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f22001378e0 con 0x7f2200142690 2024-03-29T11:23:59.742 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:59.752+0000 7f2207a0f640 1 librados: init done 2024-03-29T11:23:59.742 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:59.752+0000 7f21f5ffb640 1 -- 172.21.15.84:0/1679283077 <== mon.1 v1:172.21.15.179:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f21fc004a90 con 0x7f2200142690 2024-03-29T11:23:59.743 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:59.752+0000 7f21f5ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:23:59.743 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:59.752+0000 7f21f5ffb640 1 -- 172.21.15.84:0/1679283077 <== mon.1 v1:172.21.15.179:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4236868694 0 0) 0x7f21fc004ee0 con 0x7f2200142690 2024-03-29T11:23:59.743 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:59.752+0000 7f21f5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:23:59.743 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:59.752+0000 7f21f5ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:23:59.743 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:59.752+0000 7f21f5ffb640 10 monclient: dump: 2024-03-29T11:23:59.743 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T11:23:59.743 INFO:teuthology.misc.health.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:23:59.743 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T11:18:53.864025+0000 2024-03-29T11:23:59.743 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T11:18:53.864025+0000 2024-03-29T11:23:59.743 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T11:23:59.743 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:23:59.743 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T11:23:59.743 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T11:23:59.743 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T11:23:59.743 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:23:59.744 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:59.752+0000 7f21f77fe640 4 set_mon_vals no callback set 2024-03-29T11:23:59.744 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:59.753+0000 7f2207a0f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:23:59.744 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:59.753+0000 7f2207a0f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:23:59.744 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:59.753+0000 7f2207a0f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-29T11:23:59.744 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:59.753+0000 7f2207a0f640 1 -- 172.21.15.84:0/1679283077 --> v1:172.21.15.179:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f21e0005080 con 0x7f2200142690 2024-03-29T11:23:59.744 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:59.753+0000 7f21f5ffb640 1 -- 172.21.15.84:0/1679283077 <== mon.1 v1:172.21.15.179:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4033942909 0 0) 0x7f21fc005040 con 0x7f2200142690 2024-03-29T11:23:59.744 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:59.754+0000 7f21f5ffb640 1 -- 172.21.15.84:0/1679283077 <== mon.1 v1:172.21.15.179:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2808139587 0 0) 0x7f21fc084c40 con 0x7f2200142690 2024-03-29T11:23:59.744 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:59.754+0000 7f21f5ffb640 10 client.4360.objecter ms_dispatch 0x7f2200000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:23:59.744 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:59.754+0000 7f21f5ffb640 3 client.4360.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:23:59.744 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:59.754+0000 7f21f5ffb640 3 client.4360.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:23:59.744 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:59.754+0000 7f21f5ffb640 20 client.4360.objecter dump_active .. 0 homeless 2024-03-29T11:23:59.744 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:59.754+0000 7f22057ab640 10 monclient: get_auth_request con 0x7f21e406dc90 auth_method 0 2024-03-29T11:23:59.744 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:59.755+0000 7f21f5ffb640 10 client.4360.objecter ms_handle_connect 0x7f21e406dc90 2024-03-29T11:23:59.745 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:59.762+0000 7f21f5ffb640 1 -- 172.21.15.84:0/1679283077 <== mon.1 v1:172.21.15.179:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f21fc050520 con 0x7f2200142690 2024-03-29T11:23:59.745 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:59.762+0000 7f21f5ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:23:59.745 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:23:59.762+0000 7f21f5ffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:24:00.110 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:00.129+0000 7f2207a0f640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:24:00.110 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:00.129+0000 7f2207a0f640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:24:00.110 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:00.129+0000 7f2207a0f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-29T11:24:00.110 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:00.129+0000 7f2207a0f640 1 -- 172.21.15.84:0/1679283077 --> v1:172.21.15.179:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21e0005330 con 0x7f2200142690 2024-03-29T11:24:00.113 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:00.132+0000 7f21f5ffb640 1 -- 172.21.15.84:0/1679283077 <== mon.1 v1:172.21.15.179:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4209477549) 0x7f21fc052450 con 0x7f2200142690 2024-03-29T11:24:00.113 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:00.132+0000 7f21f5ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:24:00.113 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:00.132+0000 7f21f5ffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:24:00.113 INFO:teuthology.misc.health.smithi084.stdout: 2024-03-29T11:24:00.113 INFO:teuthology.misc.health.smithi084.stdout:{"epoch":10,"fsid":"2c3e002f-9eb0-4860-853b-4ef1ef562213","created":"2024-03-29T11:19:32.029169+0000","modified":"2024-03-29T11:19:34.200299+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:34.200299+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"5c39e276-2034-4716-bbc0-57a634eefa92","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"cd580e99-109f-4bf1-88c0-a5987bc8b9ed","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"11e60a5a-051d-453f-a190-66387cb35592","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"2c329fbe-bbc4-4030-bf87-d711cad04679","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f4213d4a-bd36-4433-91ae-d7e2b02481d3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"f0cf44ea-6055-48af-805b-aed858102ff9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"d895eaa0-cc0b-4ee4-9b5a-2dfcd89a60b7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"9bc5a7bc-8f52-4db9-96d6-de5e68d52eaa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:24:00.116 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:00.136+0000 7f2207a0f640 10 librados: watch_flush enter 2024-03-29T11:24:00.117 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:00.136+0000 7f2207a0f640 10 librados: watch_flush exit 2024-03-29T11:24:00.117 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:00.136+0000 7f2207a0f640 20 client.4360.objecter shutdown clearing up homeless session... 2024-03-29T11:24:00.117 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:00.136+0000 7f2207a0f640 10 client.4360.objecter successfully canceled tick 2024-03-29T11:24:00.117 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:00.136+0000 7f2207a0f640 1 -- 172.21.15.84:0/1679283077 >> v1:172.21.15.179:6816/2981134980 conn(0x7f21e406dc90 legacy=0x7f21e4070150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:00.117 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:00.136+0000 7f2207a0f640 10 monclient: shutdown 2024-03-29T11:24:00.117 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:00.136+0000 7f2207a0f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:24:00.117 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:00.136+0000 7f2207a0f640 1 -- 172.21.15.84:0/1679283077 >> v1:172.21.15.179:6789/0 conn(0x7f2200142690 legacy=0x7f22000acf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:00.118 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:00.137+0000 7f2207a0f640 1 -- 172.21.15.84:0/1679283077 shutdown_connections 2024-03-29T11:24:00.118 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:00.137+0000 7f2207a0f640 1 -- 172.21.15.84:0/1679283077 >> 172.21.15.84:0/1679283077 conn(0x7f2200066a00 msgr2=0x7f220013d8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:24:00.118 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:00.137+0000 7f2207a0f640 1 -- 172.21.15.84:0/1679283077 shutdown_connections 2024-03-29T11:24:00.118 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:00.137+0000 7f2207a0f640 1 -- 172.21.15.84:0/1679283077 wait complete. 2024-03-29T11:24:00.118 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:00.137+0000 7f2207a0f640 1 librados: shutdown 2024-03-29T11:24:00.131 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:24:06.938 DEBUG:teuthology.orchestra.run.smithi084:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:24:07.081 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:07.100+0000 7f99ada9d640 10 monclient: get_monmap_and_config 2024-03-29T11:24:07.081 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:07.100+0000 7f99ada9d640 10 monclient: build_initial_monmap 2024-03-29T11:24:07.082 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:07.100+0000 7f99ada9d640 10 monclient: monmap: 2024-03-29T11:24:07.082 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-29T11:24:07.082 INFO:teuthology.misc.health.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:24:07.082 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T11:24:07.101789+0000 2024-03-29T11:24:07.082 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T11:24:07.101789+0000 2024-03-29T11:24:07.082 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-29T11:24:07.082 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:24:07.082 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-29T11:24:07.082 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.noname-c 2024-03-29T11:24:07.082 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-b 2024-03-29T11:24:07.082 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:24:07.084 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:07.101+0000 7f99ada9d640 1 Processor -- start 2024-03-29T11:24:07.084 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:07.101+0000 7f99ada9d640 1 -- start start 2024-03-29T11:24:07.084 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:07.101+0000 7f99ada9d640 10 monclient: init 2024-03-29T11:24:07.084 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:07.102+0000 7f99ada9d640 10 monclient: _reopen_session rank -1 2024-03-29T11:24:07.084 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:07.102+0000 7f99ada9d640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:24:07.085 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:07.102+0000 7f99ada9d640 10 monclient(hunting): picked mon.noname-b con 0x7f99a81457f0 addr v1:172.21.15.84:6790/0 2024-03-29T11:24:07.085 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:07.102+0000 7f99ada9d640 10 monclient(hunting): picked mon.noname-a con 0x7f99a813f650 addr v1:172.21.15.84:6789/0 2024-03-29T11:24:07.085 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:07.102+0000 7f99ada9d640 10 monclient(hunting): picked mon.noname-c con 0x7f99a80a9d70 addr v1:172.21.15.179:6789/0 2024-03-29T11:24:07.085 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:07.102+0000 7f99ada9d640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f99a80aaef0 con 0x7f99a813f650 2024-03-29T11:24:07.085 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:07.102+0000 7f99ada9d640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f99a8145bd0 con 0x7f99a80a9d70 2024-03-29T11:24:07.085 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:07.102+0000 7f99ada9d640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f99a8145eb0 con 0x7f99a81457f0 2024-03-29T11:24:07.085 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:07.102+0000 7f99ada9d640 10 monclient(hunting): _renew_subs 2024-03-29T11:24:07.085 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:07.102+0000 7f99ada9d640 10 monclient(hunting): authenticate will time out at 1108.396859s 2024-03-29T11:24:07.085 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:07.103+0000 7f99a77fe640 1 --1- >> v1:172.21.15.84:6790/0 conn(0x7f99a81457f0 0x7f99a813ef40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.84:6790/0 says I am v1:172.21.15.84:37790/0 (socket says 172.21.15.84:37790) 2024-03-29T11:24:07.085 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:07.103+0000 7f99a77fe640 1 -- 172.21.15.84:0/3634815126 learned_addr learned my addr 172.21.15.84:0/3634815126 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-29T11:24:07.085 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:07.103+0000 7f99a67fc640 1 -- 172.21.15.84:0/3634815126 <== mon.1 v1:172.21.15.179:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4043163609 0 0) 0x7f99a8145bd0 con 0x7f99a80a9d70 2024-03-29T11:24:07.085 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:07.103+0000 7f99a67fc640 10 monclient(hunting): _negotiate 2024-03-29T11:24:07.085 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:07.103+0000 7f99a67fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:07.085 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:07.103+0000 7f99a67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:07.086 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:07.103+0000 7f99a67fc640 10 monclient(hunting): my global_id is 4363 2024-03-29T11:24:07.086 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:07.103+0000 7f99a67fc640 1 -- 172.21.15.84:0/3634815126 --> v1:172.21.15.179:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9990003520 con 0x7f99a80a9d70 2024-03-29T11:24:07.088 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:07.104+0000 7f99a67fc640 1 -- 172.21.15.84:0/3634815126 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1847214788 0 0) 0x7f99a8145eb0 con 0x7f99a81457f0 2024-03-29T11:24:07.088 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:07.104+0000 7f99a67fc640 10 monclient(hunting): _negotiate 2024-03-29T11:24:07.088 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:07.104+0000 7f99a67fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:07.088 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:07.104+0000 7f99a67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:07.088 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:07.104+0000 7f99a67fc640 10 monclient(hunting): my global_id is 4397 2024-03-29T11:24:07.088 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:07.104+0000 7f99a67fc640 1 -- 172.21.15.84:0/3634815126 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f99a8145bd0 con 0x7f99a81457f0 2024-03-29T11:24:07.088 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:07.104+0000 7f99a67fc640 1 -- 172.21.15.84:0/3634815126 <== mon.0 v1:172.21.15.84:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 26080660 0 0) 0x7f99a80aaef0 con 0x7f99a813f650 2024-03-29T11:24:07.088 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:07.104+0000 7f99a67fc640 10 monclient(hunting): _negotiate 2024-03-29T11:24:07.088 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:07.104+0000 7f99a67fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:07.089 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:07.104+0000 7f99a67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:07.089 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:07.104+0000 7f99a67fc640 10 monclient(hunting): my global_id is 4410 2024-03-29T11:24:07.089 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:07.104+0000 7f99a67fc640 1 -- 172.21.15.84:0/3634815126 --> v1:172.21.15.84:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f99a8145eb0 con 0x7f99a813f650 2024-03-29T11:24:07.089 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:07.104+0000 7f99a67fc640 1 -- 172.21.15.84:0/3634815126 <== mon.1 v1:172.21.15.179:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2137495285 0 0) 0x7f9990003520 con 0x7f99a80a9d70 2024-03-29T11:24:07.089 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:07.104+0000 7f99a67fc640 1 -- 172.21.15.84:0/3634815126 >> v1:172.21.15.84:6790/0 conn(0x7f99a81457f0 legacy=0x7f99a813ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:07.089 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:07.104+0000 7f99a67fc640 1 -- 172.21.15.84:0/3634815126 >> v1:172.21.15.84:6789/0 conn(0x7f99a813f650 legacy=0x7f99a80b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:07.089 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:07.104+0000 7f99a67fc640 10 monclient: _finish_hunting 0 2024-03-29T11:24:07.089 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:07.104+0000 7f99a67fc640 1 monclient: found mon.noname-c 2024-03-29T11:24:07.089 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:07.104+0000 7f99a67fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:24:07.089 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:07.104+0000 7f99a67fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.179:6789/0 2024-03-29T11:24:07.089 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:07.104+0000 7f99a67fc640 1 -- 172.21.15.84:0/3634815126 --> v1:172.21.15.179:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99a8146340 con 0x7f99a80a9d70 2024-03-29T11:24:07.089 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:07.104+0000 7f99a67fc640 10 monclient: _finish_auth 0 2024-03-29T11:24:07.089 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:07.104+0000 7f99a67fc640 10 monclient: _check_auth_tickets 2024-03-29T11:24:07.089 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:07.104+0000 7f99a67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:24:07.089 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:07.104+0000 7f99a67fc640 1 -- 172.21.15.84:0/3634815126 <== mon.1 v1:172.21.15.179:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4236868694 0 0) 0x7f99a00018c0 con 0x7f99a80a9d70 2024-03-29T11:24:07.090 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:07.104+0000 7f99ada9d640 5 monclient: authenticate success, global_id 4363 2024-03-29T11:24:07.090 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:07.104+0000 7f99ada9d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:24:07.090 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:07.104+0000 7f99a67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:07.090 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:07.104+0000 7f99a67fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:24:07.090 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:07.104+0000 7f99a67fc640 10 monclient: dump: 2024-03-29T11:24:07.090 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T11:24:07.090 INFO:teuthology.misc.health.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:24:07.090 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T11:18:53.864025+0000 2024-03-29T11:24:07.090 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T11:18:53.864025+0000 2024-03-29T11:24:07.090 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:07.090 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:24:07.090 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T11:24:07.090 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T11:24:07.090 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T11:24:07.091 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:24:07.091 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:07.104+0000 7f99ada9d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:24:07.091 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:07.105+0000 7f99a67fc640 1 -- 172.21.15.84:0/3634815126 <== mon.1 v1:172.21.15.179:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f99a0001ae0 con 0x7f99a80a9d70 2024-03-29T11:24:07.091 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:07.105+0000 7f99a67fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:24:07.091 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:07.105+0000 7f99ada9d640 10 monclient: get_monmap_and_config success 2024-03-29T11:24:07.091 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:07.105+0000 7f99ada9d640 4 set_mon_vals no callback set 2024-03-29T11:24:07.091 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:07.105+0000 7f99a67fc640 1 -- 172.21.15.84:0/3634815126 <== mon.1 v1:172.21.15.179:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4236868694 0 0) 0x7f99a0001e70 con 0x7f99a80a9d70 2024-03-29T11:24:07.091 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:07.105+0000 7f99a67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:07.091 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:07.105+0000 7f99a67fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:24:07.091 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:07.105+0000 7f99a67fc640 10 monclient: dump: 2024-03-29T11:24:07.091 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T11:24:07.091 INFO:teuthology.misc.health.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:24:07.091 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T11:18:53.864025+0000 2024-03-29T11:24:07.091 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T11:18:53.864025+0000 2024-03-29T11:24:07.091 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:07.092 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:24:07.092 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T11:24:07.092 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T11:24:07.092 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T11:24:07.092 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:24:07.092 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:07.105+0000 7f99ada9d640 10 monclient: shutdown 2024-03-29T11:24:07.092 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:07.105+0000 7f99ada9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:24:07.092 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:07.105+0000 7f99ada9d640 1 -- 172.21.15.84:0/3634815126 >> v1:172.21.15.179:6789/0 conn(0x7f99a80a9d70 legacy=0x7f99a814d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:07.092 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:07.106+0000 7f99ada9d640 1 -- 172.21.15.84:0/3634815126 shutdown_connections 2024-03-29T11:24:07.092 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:07.106+0000 7f99ada9d640 1 -- 172.21.15.84:0/3634815126 >> 172.21.15.84:0/3634815126 conn(0x7f99a8066a00 msgr2=0x7f99a8066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:24:07.092 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:07.106+0000 7f99ada9d640 1 -- 172.21.15.84:0/3634815126 shutdown_connections 2024-03-29T11:24:07.092 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:07.106+0000 7f99ada9d640 1 -- 172.21.15.84:0/3634815126 wait complete. 2024-03-29T11:24:07.092 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:07.107+0000 7f99ada9d640 10 monclient: build_initial_monmap 2024-03-29T11:24:07.092 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:07.107+0000 7f99ada9d640 10 monclient: monmap: 2024-03-29T11:24:07.092 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-29T11:24:07.093 INFO:teuthology.misc.health.smithi084.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:24:07.093 INFO:teuthology.misc.health.smithi084.stderr:last_changed 0.000000 2024-03-29T11:24:07.093 INFO:teuthology.misc.health.smithi084.stderr:created 0.000000 2024-03-29T11:24:07.093 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-29T11:24:07.093 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:24:07.093 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-29T11:24:07.093 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.noname-b 2024-03-29T11:24:07.093 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-c 2024-03-29T11:24:07.093 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:24:07.093 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:07.107+0000 7f99ada9d640 1 librados: starting msgr at 2024-03-29T11:24:07.093 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:07.107+0000 7f99ada9d640 1 librados: starting objecter 2024-03-29T11:24:07.093 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:07.107+0000 7f99ada9d640 1 Processor -- start 2024-03-29T11:24:07.102 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:07.108+0000 7f99ada9d640 1 -- start start 2024-03-29T11:24:07.102 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:07.108+0000 7f99ada9d640 1 librados: setting wanted keys 2024-03-29T11:24:07.102 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:07.108+0000 7f99ada9d640 1 librados: calling monclient init 2024-03-29T11:24:07.102 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:07.108+0000 7f99ada9d640 10 monclient: init 2024-03-29T11:24:07.102 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:07.109+0000 7f99ada9d640 10 monclient: _reopen_session rank -1 2024-03-29T11:24:07.102 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:07.109+0000 7f99ada9d640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:24:07.102 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:07.109+0000 7f99ada9d640 10 monclient(hunting): picked mon.noname-a con 0x7f99a80a9d70 addr v1:172.21.15.84:6789/0 2024-03-29T11:24:07.102 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:07.109+0000 7f99ada9d640 10 monclient(hunting): picked mon.noname-c con 0x7f99a813f650 addr v1:172.21.15.84:6790/0 2024-03-29T11:24:07.102 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:07.109+0000 7f99a77fe640 1 --1- >> v1:172.21.15.84:6789/0 conn(0x7f99a80a9d70 0x7f99a81dba10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.84:6789/0 says I am v1:172.21.15.84:44414/0 (socket says 172.21.15.84:44414) 2024-03-29T11:24:07.102 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:07.109+0000 7f99ada9d640 10 monclient(hunting): picked mon.noname-b con 0x7f99a81457f0 addr v1:172.21.15.179:6789/0 2024-03-29T11:24:07.102 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:07.109+0000 7f99ada9d640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f99a81445c0 con 0x7f99a80a9d70 2024-03-29T11:24:07.102 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:07.109+0000 7f99ada9d640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f99a81dc120 con 0x7f99a81457f0 2024-03-29T11:24:07.102 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:07.109+0000 7f99ada9d640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f99a81381a0 con 0x7f99a813f650 2024-03-29T11:24:07.103 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:07.109+0000 7f99ada9d640 10 monclient(hunting): _renew_subs 2024-03-29T11:24:07.103 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:07.109+0000 7f99ada9d640 10 monclient(hunting): authenticate will time out at 1108.403780s 2024-03-29T11:24:07.103 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:07.109+0000 7f99a77fe640 1 -- 172.21.15.84:0/142873083 learned_addr learned my addr 172.21.15.84:0/142873083 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-29T11:24:07.103 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:07.110+0000 7f99a77fe640 1 -- 172.21.15.84:0/142873083 >> v1:172.21.15.84:6789/0 conn(0x7f99a80a9d70 legacy=0x7f99a81dba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-03-29T11:24:07.103 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:07.110+0000 7f99a77fe640 1 -- 172.21.15.84:0/142873083 >> v1:172.21.15.84:6789/0 conn(0x7f99a80a9d70 legacy=0x7f99a81dba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-29T11:24:07.103 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:07.110+0000 7f99a77fe640 1 --1- 172.21.15.84:0/142873083 >> v1:172.21.15.84:6789/0 conn(0x7f99a80a9d70 0x7f99a81dba10 :-1 s=OPENED pgs=468165 cs=1 l=1).handle_message read tag failed 2024-03-29T11:24:07.103 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:07.110+0000 7f99a77fe640 1 --1- 172.21.15.84:0/142873083 >> v1:172.21.15.84:6789/0 conn(0x7f99a80a9d70 0x7f99a81dba10 :-1 s=OPENED pgs=468165 cs=1 l=1).fault on lossy channel, failing 2024-03-29T11:24:07.103 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:07.110+0000 7f998ffff640 10 client.?.objecter ms_handle_connect 0x7f99a80a9d70 2024-03-29T11:24:07.103 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:07.110+0000 7f998ffff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:24:07.103 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:07.110+0000 7f998ffff640 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.84:6789/0 2024-03-29T11:24:07.103 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:07.110+0000 7f998ffff640 10 client.?.objecter ms_handle_connect 0x7f99a81457f0 2024-03-29T11:24:07.103 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:07.110+0000 7f998ffff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:24:07.103 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:07.110+0000 7f998ffff640 10 client.?.objecter ms_handle_connect 0x7f99a813f650 2024-03-29T11:24:07.103 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:07.110+0000 7f998ffff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:24:07.104 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:07.110+0000 7f998ffff640 1 -- 172.21.15.84:0/142873083 <== mon.1 v1:172.21.15.179:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4149630573 0 0) 0x7f99a81dc120 con 0x7f99a81457f0 2024-03-29T11:24:07.104 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:07.110+0000 7f998ffff640 10 monclient(hunting): _negotiate 2024-03-29T11:24:07.104 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:07.111+0000 7f998ffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:07.104 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:07.111+0000 7f998ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:07.104 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:07.111+0000 7f998ffff640 10 monclient(hunting): my global_id is 4366 2024-03-29T11:24:07.104 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:07.111+0000 7f998ffff640 1 -- 172.21.15.84:0/142873083 --> v1:172.21.15.179:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9994004eb0 con 0x7f99a81457f0 2024-03-29T11:24:07.104 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:07.111+0000 7f998ffff640 1 -- 172.21.15.84:0/142873083 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 597921141 0 0) 0x7f99a81381a0 con 0x7f99a813f650 2024-03-29T11:24:07.104 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:07.111+0000 7f998ffff640 10 monclient(hunting): _negotiate 2024-03-29T11:24:07.104 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:07.111+0000 7f998ffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:07.104 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:07.111+0000 7f998ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:07.104 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:07.111+0000 7f998ffff640 10 monclient(hunting): my global_id is 4400 2024-03-29T11:24:07.104 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:07.111+0000 7f998ffff640 1 -- 172.21.15.84:0/142873083 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f99a81dc120 con 0x7f99a813f650 2024-03-29T11:24:07.104 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:07.111+0000 7f998ffff640 1 -- 172.21.15.84:0/142873083 <== mon.1 v1:172.21.15.179:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1652326791 0 0) 0x7f9994004eb0 con 0x7f99a81457f0 2024-03-29T11:24:07.104 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:07.111+0000 7f998ffff640 1 -- 172.21.15.84:0/142873083 --> v1:172.21.15.179:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f99a81381a0 con 0x7f99a81457f0 2024-03-29T11:24:07.105 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:07.111+0000 7f998ffff640 1 -- 172.21.15.84:0/142873083 <== mon.2 v1:172.21.15.84:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 937164524 0 0) 0x7f99a81dc120 con 0x7f99a813f650 2024-03-29T11:24:07.105 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:07.111+0000 7f998ffff640 1 -- 172.21.15.84:0/142873083 --> v1:172.21.15.84:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9994004eb0 con 0x7f99a813f650 2024-03-29T11:24:07.105 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:07.111+0000 7f998ffff640 1 -- 172.21.15.84:0/142873083 <== mon.1 v1:172.21.15.179:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4236868694 0 0) 0x7f99a0001800 con 0x7f99a81457f0 2024-03-29T11:24:07.105 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:07.112+0000 7f998ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:07.105 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:07.112+0000 7f998ffff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:24:07.105 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:07.112+0000 7f998ffff640 10 monclient(hunting): dump: 2024-03-29T11:24:07.105 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T11:24:07.105 INFO:teuthology.misc.health.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:24:07.105 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T11:18:53.864025+0000 2024-03-29T11:24:07.105 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T11:18:53.864025+0000 2024-03-29T11:24:07.105 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:07.105 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:24:07.105 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T11:24:07.105 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T11:24:07.105 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T11:24:07.106 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:24:07.106 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:07.112+0000 7f998ffff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:24:07.106 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:07.112+0000 7f998ffff640 1 -- 172.21.15.84:0/142873083 <== mon.2 v1:172.21.15.84:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4236868694 0 0) 0x7f9998001870 con 0x7f99a813f650 2024-03-29T11:24:07.106 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:07.112+0000 7f998ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:07.106 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:07.112+0000 7f998ffff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:24:07.106 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:07.112+0000 7f998ffff640 10 monclient(hunting): dump: 2024-03-29T11:24:07.106 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T11:24:07.106 INFO:teuthology.misc.health.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:24:07.106 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T11:18:53.864025+0000 2024-03-29T11:24:07.106 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T11:18:53.864025+0000 2024-03-29T11:24:07.106 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:07.106 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:24:07.106 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T11:24:07.106 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T11:24:07.106 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T11:24:07.107 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:24:07.107 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:07.112+0000 7f998ffff640 1 -- 172.21.15.84:0/142873083 <== mon.2 v1:172.21.15.84:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3553282511 0 0) 0x7f9994004eb0 con 0x7f99a813f650 2024-03-29T11:24:07.107 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:07.112+0000 7f998ffff640 1 -- 172.21.15.84:0/142873083 >> v1:172.21.15.179:6789/0 conn(0x7f99a81457f0 legacy=0x7f99a81e7250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:07.107 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:07.112+0000 7f998ffff640 1 -- 172.21.15.84:0/142873083 >> v1:172.21.15.84:6789/0 conn(0x7f99a80a9d70 legacy=0x7f99a81dba10 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-29T11:24:07.107 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:07.112+0000 7f998ffff640 10 monclient: _finish_hunting 0 2024-03-29T11:24:07.107 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:07.112+0000 7f998ffff640 1 monclient: found mon.c 2024-03-29T11:24:07.107 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:07.112+0000 7f998ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:24:07.107 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:07.112+0000 7f998ffff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-29T11:24:07.107 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:07.112+0000 7f998ffff640 1 -- 172.21.15.84:0/142873083 --> v1:172.21.15.84:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99a8138330 con 0x7f99a813f650 2024-03-29T11:24:07.107 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:07.112+0000 7f998ffff640 10 monclient: _finish_auth 0 2024-03-29T11:24:07.107 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:07.112+0000 7f998ffff640 10 monclient: _check_auth_tickets 2024-03-29T11:24:07.107 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:07.112+0000 7f998ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:24:07.107 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:07.112+0000 7f99ada9d640 5 monclient: authenticate success, global_id 4400 2024-03-29T11:24:07.108 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:07.112+0000 7f99ada9d640 10 monclient: _renew_subs 2024-03-29T11:24:07.108 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:07.112+0000 7f99ada9d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-29T11:24:07.108 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:07.112+0000 7f99ada9d640 1 -- 172.21.15.84:0/142873083 --> v1:172.21.15.84:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f99a8144da0 con 0x7f99a813f650 2024-03-29T11:24:07.108 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:07.112+0000 7f99ada9d640 10 client.4400.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:24:07.108 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:07.112+0000 7f99ada9d640 10 monclient: _renew_subs 2024-03-29T11:24:07.108 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:07.112+0000 7f99ada9d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-29T11:24:07.108 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:07.112+0000 7f99ada9d640 1 -- 172.21.15.84:0/142873083 --> v1:172.21.15.84:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f99a8140180 con 0x7f99a813f650 2024-03-29T11:24:07.108 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:07.112+0000 7f99ada9d640 1 librados: init done 2024-03-29T11:24:07.108 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:07.113+0000 7f998ffff640 1 -- 172.21.15.84:0/142873083 <== mon.2 v1:172.21.15.84:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9998001d40 con 0x7f99a813f650 2024-03-29T11:24:07.108 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:07.113+0000 7f998ffff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:24:07.108 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:07.113+0000 7f99a57fa640 4 set_mon_vals no callback set 2024-03-29T11:24:07.108 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:07.114+0000 7f998ffff640 1 -- 172.21.15.84:0/142873083 <== mon.2 v1:172.21.15.84:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4236868694 0 0) 0x7f99980037b0 con 0x7f99a813f650 2024-03-29T11:24:07.108 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:07.114+0000 7f998ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:07.108 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:07.114+0000 7f998ffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:24:07.108 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:07.114+0000 7f998ffff640 10 monclient: dump: 2024-03-29T11:24:07.109 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T11:24:07.109 INFO:teuthology.misc.health.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:24:07.109 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T11:18:53.864025+0000 2024-03-29T11:24:07.109 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T11:18:53.864025+0000 2024-03-29T11:24:07.109 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:07.109 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:24:07.109 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T11:24:07.109 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T11:24:07.109 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T11:24:07.109 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:24:07.109 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:07.114+0000 7f99ada9d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:24:07.109 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:07.114+0000 7f99ada9d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:24:07.109 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:07.114+0000 7f99ada9d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-29T11:24:07.109 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:07.114+0000 7f99ada9d640 1 -- 172.21.15.84:0/142873083 --> v1:172.21.15.84:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9988005080 con 0x7f99a813f650 2024-03-29T11:24:07.109 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:07.116+0000 7f998ffff640 1 -- 172.21.15.84:0/142873083 <== mon.2 v1:172.21.15.84:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4033942909 0 0) 0x7f9998003910 con 0x7f99a813f650 2024-03-29T11:24:07.110 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:07.117+0000 7f998ffff640 1 -- 172.21.15.84:0/142873083 <== mon.2 v1:172.21.15.84:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2808139587 0 0) 0x7f9998083640 con 0x7f99a813f650 2024-03-29T11:24:07.110 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:07.117+0000 7f998ffff640 10 client.4400.objecter ms_dispatch 0x7f99a8000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:24:07.110 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:07.117+0000 7f998ffff640 3 client.4400.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:24:07.110 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:07.117+0000 7f998ffff640 3 client.4400.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:24:07.110 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:07.117+0000 7f998ffff640 20 client.4400.objecter dump_active .. 0 homeless 2024-03-29T11:24:07.110 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:07.117+0000 7f99a77fe640 10 monclient: get_auth_request con 0x7f999406df60 auth_method 0 2024-03-29T11:24:07.110 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:07.117+0000 7f998ffff640 10 client.4400.objecter ms_handle_connect 0x7f999406df60 2024-03-29T11:24:07.110 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:07.122+0000 7f998ffff640 1 -- 172.21.15.84:0/142873083 <== mon.2 v1:172.21.15.84:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f999804ef20 con 0x7f99a813f650 2024-03-29T11:24:07.110 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:07.122+0000 7f998ffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:24:07.111 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:07.122+0000 7f998ffff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:24:07.473 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:07.492+0000 7f99ada9d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:24:07.473 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:07.492+0000 7f99ada9d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:24:07.473 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:07.492+0000 7f99ada9d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-29T11:24:07.473 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:07.492+0000 7f99ada9d640 1 -- 172.21.15.84:0/142873083 --> v1:172.21.15.84:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9988005330 con 0x7f99a813f650 2024-03-29T11:24:07.474 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:07.493+0000 7f998ffff640 1 -- 172.21.15.84:0/142873083 <== mon.2 v1:172.21.15.84:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4209477549) 0x7f9998050e50 con 0x7f99a813f650 2024-03-29T11:24:07.474 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:07.493+0000 7f998ffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:24:07.474 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:07.493+0000 7f998ffff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:24:07.474 INFO:teuthology.misc.health.smithi084.stdout: 2024-03-29T11:24:07.474 INFO:teuthology.misc.health.smithi084.stdout:{"epoch":10,"fsid":"2c3e002f-9eb0-4860-853b-4ef1ef562213","created":"2024-03-29T11:19:32.029169+0000","modified":"2024-03-29T11:19:34.200299+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:34.200299+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"5c39e276-2034-4716-bbc0-57a634eefa92","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"cd580e99-109f-4bf1-88c0-a5987bc8b9ed","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"11e60a5a-051d-453f-a190-66387cb35592","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"2c329fbe-bbc4-4030-bf87-d711cad04679","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f4213d4a-bd36-4433-91ae-d7e2b02481d3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"f0cf44ea-6055-48af-805b-aed858102ff9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"d895eaa0-cc0b-4ee4-9b5a-2dfcd89a60b7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"9bc5a7bc-8f52-4db9-96d6-de5e68d52eaa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:24:07.477 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:07.496+0000 7f99ada9d640 10 librados: watch_flush enter 2024-03-29T11:24:07.477 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:07.497+0000 7f99ada9d640 10 librados: watch_flush exit 2024-03-29T11:24:07.477 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:07.497+0000 7f99ada9d640 20 client.4400.objecter shutdown clearing up homeless session... 2024-03-29T11:24:07.477 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:07.497+0000 7f99ada9d640 10 client.4400.objecter successfully canceled tick 2024-03-29T11:24:07.478 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:07.497+0000 7f99ada9d640 1 -- 172.21.15.84:0/142873083 >> v1:172.21.15.179:6816/2981134980 conn(0x7f999406df60 legacy=0x7f9994070420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:07.478 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:07.497+0000 7f99ada9d640 10 monclient: shutdown 2024-03-29T11:24:07.478 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:07.497+0000 7f99ada9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:24:07.478 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:07.497+0000 7f99ada9d640 1 -- 172.21.15.84:0/142873083 >> v1:172.21.15.84:6790/0 conn(0x7f99a813f650 legacy=0x7f99a81374e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:07.478 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:07.497+0000 7f99ada9d640 1 -- 172.21.15.84:0/142873083 shutdown_connections 2024-03-29T11:24:07.478 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:07.497+0000 7f99ada9d640 1 -- 172.21.15.84:0/142873083 >> 172.21.15.84:0/142873083 conn(0x7f99a8066a00 msgr2=0x7f99a80b3a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:24:07.478 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:07.497+0000 7f99ada9d640 1 -- 172.21.15.84:0/142873083 shutdown_connections 2024-03-29T11:24:07.478 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:07.497+0000 7f99ada9d640 1 -- 172.21.15.84:0/142873083 wait complete. 2024-03-29T11:24:07.478 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:07.497+0000 7f99ada9d640 1 librados: shutdown 2024-03-29T11:24:07.492 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:24:14.299 DEBUG:teuthology.orchestra.run.smithi084:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:24:14.448 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:14.467+0000 7fa297723640 10 monclient: get_monmap_and_config 2024-03-29T11:24:14.449 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:14.467+0000 7fa297723640 10 monclient: build_initial_monmap 2024-03-29T11:24:14.449 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:14.467+0000 7fa297723640 10 monclient: monmap: 2024-03-29T11:24:14.449 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-29T11:24:14.449 INFO:teuthology.misc.health.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:24:14.449 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T11:24:14.468533+0000 2024-03-29T11:24:14.449 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T11:24:14.468533+0000 2024-03-29T11:24:14.449 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-29T11:24:14.449 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:24:14.449 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-29T11:24:14.449 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.noname-c 2024-03-29T11:24:14.449 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-b 2024-03-29T11:24:14.449 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:24:14.450 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:14.468+0000 7fa297723640 1 Processor -- start 2024-03-29T11:24:14.450 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:14.468+0000 7fa297723640 1 -- start start 2024-03-29T11:24:14.450 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:14.468+0000 7fa297723640 10 monclient: init 2024-03-29T11:24:14.451 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:14.470+0000 7fa297723640 10 monclient: _reopen_session rank -1 2024-03-29T11:24:14.451 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:14.470+0000 7fa297723640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:24:14.451 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:14.470+0000 7fa297723640 10 monclient(hunting): picked mon.noname-b con 0x7fa2880389e0 addr v1:172.21.15.84:6790/0 2024-03-29T11:24:14.451 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:14.470+0000 7fa297723640 10 monclient(hunting): picked mon.noname-c con 0x7fa288038dc0 addr v1:172.21.15.179:6789/0 2024-03-29T11:24:14.457 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:14.471+0000 7fa297723640 10 monclient(hunting): picked mon.noname-a con 0x7fa288030660 addr v1:172.21.15.84:6789/0 2024-03-29T11:24:14.457 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:14.471+0000 7fa297723640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa2880c54c0 con 0x7fa288030660 2024-03-29T11:24:14.457 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:14.471+0000 7fa297723640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa288036430 con 0x7fa288038dc0 2024-03-29T11:24:14.457 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:14.471+0000 7fa297723640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa288039840 con 0x7fa2880389e0 2024-03-29T11:24:14.457 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:14.471+0000 7fa297723640 10 monclient(hunting): _renew_subs 2024-03-29T11:24:14.457 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:14.471+0000 7fa297723640 10 monclient(hunting): authenticate will time out at 1115.765342s 2024-03-29T11:24:14.457 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:14.471+0000 7fa2954bf640 1 --1- >> v1:172.21.15.84:6790/0 conn(0x7fa2880389e0 0x7fa288033520 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.84:6790/0 says I am v1:172.21.15.84:42074/0 (socket says 172.21.15.84:42074) 2024-03-29T11:24:14.457 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:14.471+0000 7fa2954bf640 1 -- 172.21.15.84:0/2264896711 learned_addr learned my addr 172.21.15.84:0/2264896711 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-29T11:24:14.457 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:14.474+0000 7fa287fff640 1 -- 172.21.15.84:0/2264896711 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3260767091 0 0) 0x7fa288039840 con 0x7fa2880389e0 2024-03-29T11:24:14.457 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:14.474+0000 7fa287fff640 10 monclient(hunting): _negotiate 2024-03-29T11:24:14.457 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:14.474+0000 7fa287fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:14.458 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:14.474+0000 7fa287fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:14.458 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:14.474+0000 7fa287fff640 10 monclient(hunting): my global_id is 4403 2024-03-29T11:24:14.458 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:14.474+0000 7fa287fff640 1 -- 172.21.15.84:0/2264896711 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa27c003520 con 0x7fa2880389e0 2024-03-29T11:24:14.458 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:14.474+0000 7fa287fff640 1 -- 172.21.15.84:0/2264896711 <== mon.2 v1:172.21.15.84:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1703139348 0 0) 0x7fa27c003520 con 0x7fa2880389e0 2024-03-29T11:24:14.458 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:14.474+0000 7fa287fff640 1 -- 172.21.15.84:0/2264896711 >> v1:172.21.15.179:6789/0 conn(0x7fa288038dc0 legacy=0x7fa2880cadd0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:24:14.458 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:14.474+0000 7fa287fff640 1 -- 172.21.15.84:0/2264896711 >> v1:172.21.15.84:6789/0 conn(0x7fa288030660 legacy=0x7fa2880d3170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:14.458 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:14.474+0000 7fa287fff640 10 monclient: _finish_hunting 0 2024-03-29T11:24:14.458 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:14.474+0000 7fa287fff640 1 monclient: found mon.noname-b 2024-03-29T11:24:14.458 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:14.474+0000 7fa287fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:24:14.458 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:14.474+0000 7fa287fff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.84:6790/0 2024-03-29T11:24:14.458 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:14.474+0000 7fa287fff640 1 -- 172.21.15.84:0/2264896711 --> v1:172.21.15.84:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa288035690 con 0x7fa2880389e0 2024-03-29T11:24:14.458 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:14.475+0000 7fa287fff640 10 monclient: _finish_auth 0 2024-03-29T11:24:14.458 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:14.475+0000 7fa287fff640 10 monclient: _check_auth_tickets 2024-03-29T11:24:14.458 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:14.475+0000 7fa287fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:24:14.459 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:14.475+0000 7fa287fff640 1 -- 172.21.15.84:0/2264896711 <== mon.2 v1:172.21.15.84:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4236868694 0 0) 0x7fa28c001690 con 0x7fa2880389e0 2024-03-29T11:24:14.459 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:14.475+0000 7fa287fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:14.459 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:14.475+0000 7fa287fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:24:14.459 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:14.475+0000 7fa287fff640 10 monclient: dump: 2024-03-29T11:24:14.459 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T11:24:14.459 INFO:teuthology.misc.health.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:24:14.459 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T11:18:53.864025+0000 2024-03-29T11:24:14.459 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T11:18:53.864025+0000 2024-03-29T11:24:14.459 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:14.459 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:24:14.459 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T11:24:14.459 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T11:24:14.459 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T11:24:14.459 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:24:14.460 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:14.475+0000 7fa297723640 5 monclient: authenticate success, global_id 4403 2024-03-29T11:24:14.460 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:14.475+0000 7fa297723640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:24:14.460 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:14.475+0000 7fa287fff640 1 -- 172.21.15.84:0/2264896711 <== mon.2 v1:172.21.15.84:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa28c001920 con 0x7fa2880389e0 2024-03-29T11:24:14.460 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:14.475+0000 7fa287fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:24:14.460 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:14.475+0000 7fa287fff640 1 -- 172.21.15.84:0/2264896711 <== mon.2 v1:172.21.15.84:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4236868694 0 0) 0x7fa28c001cb0 con 0x7fa2880389e0 2024-03-29T11:24:14.460 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:14.475+0000 7fa287fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:14.460 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:14.475+0000 7fa287fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:24:14.460 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:14.475+0000 7fa287fff640 10 monclient: dump: 2024-03-29T11:24:14.460 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T11:24:14.460 INFO:teuthology.misc.health.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:24:14.460 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T11:18:53.864025+0000 2024-03-29T11:24:14.460 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T11:18:53.864025+0000 2024-03-29T11:24:14.460 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:14.460 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:24:14.460 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T11:24:14.461 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T11:24:14.461 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T11:24:14.461 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:24:14.461 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:14.475+0000 7fa297723640 10 monclient: get_monmap_and_config success 2024-03-29T11:24:14.461 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:14.475+0000 7fa297723640 4 set_mon_vals no callback set 2024-03-29T11:24:14.462 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:14.476+0000 7fa297723640 10 monclient: shutdown 2024-03-29T11:24:14.463 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:14.476+0000 7fa297723640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:24:14.463 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:14.476+0000 7fa297723640 1 -- 172.21.15.84:0/2264896711 >> v1:172.21.15.84:6790/0 conn(0x7fa2880389e0 legacy=0x7fa288033520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:14.463 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:14.476+0000 7fa297723640 1 -- 172.21.15.84:0/2264896711 shutdown_connections 2024-03-29T11:24:14.463 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:14.476+0000 7fa297723640 1 -- 172.21.15.84:0/2264896711 >> 172.21.15.84:0/2264896711 conn(0x7fa288005c00 msgr2=0x7fa288006010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:24:14.463 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:14.478+0000 7fa297723640 1 -- 172.21.15.84:0/2264896711 shutdown_connections 2024-03-29T11:24:14.463 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:14.478+0000 7fa297723640 1 -- 172.21.15.84:0/2264896711 wait complete. 2024-03-29T11:24:14.463 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:14.479+0000 7fa297723640 10 monclient: build_initial_monmap 2024-03-29T11:24:14.463 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:14.479+0000 7fa297723640 10 monclient: monmap: 2024-03-29T11:24:14.463 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-29T11:24:14.463 INFO:teuthology.misc.health.smithi084.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:24:14.463 INFO:teuthology.misc.health.smithi084.stderr:last_changed 0.000000 2024-03-29T11:24:14.463 INFO:teuthology.misc.health.smithi084.stderr:created 0.000000 2024-03-29T11:24:14.463 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-29T11:24:14.464 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:24:14.464 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-29T11:24:14.464 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.noname-b 2024-03-29T11:24:14.464 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-c 2024-03-29T11:24:14.464 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:24:14.464 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:14.480+0000 7fa297723640 1 librados: starting msgr at 2024-03-29T11:24:14.464 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:14.480+0000 7fa297723640 1 librados: starting objecter 2024-03-29T11:24:14.464 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:14.480+0000 7fa297723640 1 Processor -- start 2024-03-29T11:24:14.464 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:14.480+0000 7fa297723640 1 -- start start 2024-03-29T11:24:14.464 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:14.480+0000 7fa297723640 1 librados: setting wanted keys 2024-03-29T11:24:14.464 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:14.480+0000 7fa297723640 1 librados: calling monclient init 2024-03-29T11:24:14.464 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:14.480+0000 7fa297723640 10 monclient: init 2024-03-29T11:24:14.464 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:14.481+0000 7fa297723640 10 monclient: _reopen_session rank -1 2024-03-29T11:24:14.464 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:14.481+0000 7fa297723640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:24:14.465 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:14.482+0000 7fa297723640 10 monclient(hunting): picked mon.noname-a con 0x7fa288030660 addr v1:172.21.15.84:6789/0 2024-03-29T11:24:14.465 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:14.482+0000 7fa297723640 10 monclient(hunting): picked mon.noname-b con 0x7fa2880389e0 addr v1:172.21.15.179:6789/0 2024-03-29T11:24:14.465 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:14.482+0000 7fa297723640 10 monclient(hunting): picked mon.noname-c con 0x7fa2880c8310 addr v1:172.21.15.84:6790/0 2024-03-29T11:24:14.465 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:14.482+0000 7fa297723640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa2880c6dc0 con 0x7fa288030660 2024-03-29T11:24:14.465 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:14.482+0000 7fa297723640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa28802d150 con 0x7fa2880389e0 2024-03-29T11:24:14.465 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:14.482+0000 7fa297723640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa28802eda0 con 0x7fa2880c8310 2024-03-29T11:24:14.465 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:14.482+0000 7fa297723640 10 monclient(hunting): _renew_subs 2024-03-29T11:24:14.465 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:14.482+0000 7fa297723640 10 monclient(hunting): authenticate will time out at 1115.776615s 2024-03-29T11:24:14.467 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:14.484+0000 7fa295cc0640 1 --1- >> v1:172.21.15.84:6790/0 conn(0x7fa2880c8310 0x7fa288167460 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.84:6790/0 says I am v1:172.21.15.84:43988/0 (socket says 172.21.15.84:43988) 2024-03-29T11:24:14.467 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:14.484+0000 7fa295cc0640 1 -- 172.21.15.84:0/3167863825 learned_addr learned my addr 172.21.15.84:0/3167863825 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-29T11:24:14.467 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:14.485+0000 7fa285ffb640 10 client.?.objecter ms_handle_connect 0x7fa2880c8310 2024-03-29T11:24:14.467 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:14.485+0000 7fa285ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:24:14.468 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:14.485+0000 7fa285ffb640 10 client.?.objecter ms_handle_connect 0x7fa288030660 2024-03-29T11:24:14.468 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:14.485+0000 7fa285ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:24:14.468 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:14.485+0000 7fa285ffb640 1 -- 172.21.15.84:0/3167863825 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2401082977 0 0) 0x7fa28802eda0 con 0x7fa2880c8310 2024-03-29T11:24:14.468 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:14.485+0000 7fa285ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:24:14.468 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:14.486+0000 7fa285ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:14.468 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:14.486+0000 7fa285ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:14.468 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:14.486+0000 7fa285ffb640 10 monclient(hunting): my global_id is 4406 2024-03-29T11:24:14.468 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:14.486+0000 7fa285ffb640 1 -- 172.21.15.84:0/3167863825 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa274004eb0 con 0x7fa2880c8310 2024-03-29T11:24:14.468 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:14.486+0000 7fa285ffb640 1 -- 172.21.15.84:0/3167863825 <== mon.0 v1:172.21.15.84:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2049732679 0 0) 0x7fa288035690 con 0x7fa288030660 2024-03-29T11:24:14.468 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:14.486+0000 7fa285ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:24:14.468 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:14.486+0000 7fa285ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:14.468 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:14.486+0000 7fa285ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:14.468 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:14.486+0000 7fa285ffb640 10 monclient(hunting): my global_id is 4416 2024-03-29T11:24:14.472 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:14.487+0000 7fa285ffb640 1 -- 172.21.15.84:0/3167863825 --> v1:172.21.15.84:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa28802eda0 con 0x7fa288030660 2024-03-29T11:24:14.472 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:14.487+0000 7fa285ffb640 10 client.?.objecter ms_handle_connect 0x7fa2880389e0 2024-03-29T11:24:14.472 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:14.487+0000 7fa285ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:24:14.472 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:14.487+0000 7fa285ffb640 1 -- 172.21.15.84:0/3167863825 <== mon.2 v1:172.21.15.84:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4165296476 0 0) 0x7fa274004eb0 con 0x7fa2880c8310 2024-03-29T11:24:14.472 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:14.487+0000 7fa285ffb640 1 -- 172.21.15.84:0/3167863825 --> v1:172.21.15.84:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa288035690 con 0x7fa2880c8310 2024-03-29T11:24:14.472 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:14.487+0000 7fa285ffb640 1 -- 172.21.15.84:0/3167863825 <== mon.2 v1:172.21.15.84:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4236868694 0 0) 0x7fa29004fdd0 con 0x7fa2880c8310 2024-03-29T11:24:14.473 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:14.487+0000 7fa285ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:14.473 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:14.487+0000 7fa285ffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:24:14.473 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:14.488+0000 7fa285ffb640 10 monclient(hunting): dump: 2024-03-29T11:24:14.473 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T11:24:14.473 INFO:teuthology.misc.health.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:24:14.473 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T11:18:53.864025+0000 2024-03-29T11:24:14.473 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T11:18:53.864025+0000 2024-03-29T11:24:14.473 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:14.473 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:24:14.473 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T11:24:14.473 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T11:24:14.473 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T11:24:14.473 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:24:14.473 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:14.488+0000 7fa285ffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:24:14.474 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:14.488+0000 7fa285ffb640 1 -- 172.21.15.84:0/3167863825 <== mon.2 v1:172.21.15.84:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 6349512 0 0) 0x7fa288035690 con 0x7fa2880c8310 2024-03-29T11:24:14.474 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:14.488+0000 7fa285ffb640 1 -- 172.21.15.84:0/3167863825 >> v1:172.21.15.179:6789/0 conn(0x7fa2880389e0 legacy=0x7fa28802d5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:14.474 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:14.488+0000 7fa285ffb640 1 -- 172.21.15.84:0/3167863825 >> v1:172.21.15.84:6789/0 conn(0x7fa288030660 legacy=0x7fa28802f9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:14.474 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:14.488+0000 7fa285ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:24:14.474 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:14.488+0000 7fa285ffb640 1 monclient: found mon.c 2024-03-29T11:24:14.474 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:14.488+0000 7fa285ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:24:14.474 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:14.488+0000 7fa285ffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-29T11:24:14.474 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:14.488+0000 7fa285ffb640 1 -- 172.21.15.84:0/3167863825 --> v1:172.21.15.84:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa28802f010 con 0x7fa2880c8310 2024-03-29T11:24:14.474 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:14.488+0000 7fa285ffb640 10 monclient: _finish_auth 0 2024-03-29T11:24:14.474 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:14.488+0000 7fa285ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:24:14.474 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:14.488+0000 7fa285ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:24:14.474 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:14.488+0000 7fa285ffb640 1 -- 172.21.15.84:0/3167863825 <== mon.2 v1:172.21.15.84:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa29004f400 con 0x7fa2880c8310 2024-03-29T11:24:14.474 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:14.488+0000 7fa285ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:24:14.474 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:14.488+0000 7fa285ffb640 1 -- 172.21.15.84:0/3167863825 <== mon.2 v1:172.21.15.84:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4236868694 0 0) 0x7fa290075f10 con 0x7fa2880c8310 2024-03-29T11:24:14.475 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:14.488+0000 7fa285ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:14.475 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:14.489+0000 7fa285ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:24:14.475 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:14.489+0000 7fa285ffb640 10 monclient: dump: 2024-03-29T11:24:14.475 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T11:24:14.475 INFO:teuthology.misc.health.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:24:14.475 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T11:18:53.864025+0000 2024-03-29T11:24:14.475 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T11:18:53.864025+0000 2024-03-29T11:24:14.475 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:14.475 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:24:14.475 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T11:24:14.475 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T11:24:14.475 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T11:24:14.475 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:24:14.475 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:14.489+0000 7fa297723640 5 monclient: authenticate success, global_id 4406 2024-03-29T11:24:14.475 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:14.489+0000 7fa297723640 10 monclient: _renew_subs 2024-03-29T11:24:14.476 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:14.489+0000 7fa297723640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-29T11:24:14.476 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:14.489+0000 7fa297723640 1 -- 172.21.15.84:0/3167863825 --> v1:172.21.15.84:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa28802dcf0 con 0x7fa2880c8310 2024-03-29T11:24:14.476 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:14.489+0000 7fa297723640 10 client.4406.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:24:14.476 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:14.489+0000 7fa297723640 10 monclient: _renew_subs 2024-03-29T11:24:14.476 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:14.489+0000 7fa297723640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-29T11:24:14.476 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:14.489+0000 7fa297723640 1 -- 172.21.15.84:0/3167863825 --> v1:172.21.15.84:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa288168d60 con 0x7fa2880c8310 2024-03-29T11:24:14.476 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:14.489+0000 7fa297723640 1 librados: init done 2024-03-29T11:24:14.476 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:14.490+0000 7fa2877fe640 4 set_mon_vals no callback set 2024-03-29T11:24:14.476 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:14.491+0000 7fa285ffb640 1 -- 172.21.15.84:0/3167863825 <== mon.2 v1:172.21.15.84:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4033942909 0 0) 0x7fa290067bf0 con 0x7fa2880c8310 2024-03-29T11:24:14.476 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:14.491+0000 7fa297723640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:24:14.476 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:14.491+0000 7fa297723640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:24:14.476 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:14.491+0000 7fa297723640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-29T11:24:14.476 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:14.491+0000 7fa297723640 1 -- 172.21.15.84:0/3167863825 --> v1:172.21.15.84:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa270005080 con 0x7fa2880c8310 2024-03-29T11:24:14.477 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:14.492+0000 7fa285ffb640 1 -- 172.21.15.84:0/3167863825 <== mon.2 v1:172.21.15.84:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2808139587 0 0) 0x7fa2900ff450 con 0x7fa2880c8310 2024-03-29T11:24:14.477 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:14.492+0000 7fa285ffb640 10 client.4406.objecter ms_dispatch 0x7fa290000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:24:14.477 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:14.492+0000 7fa285ffb640 3 client.4406.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:24:14.477 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:14.492+0000 7fa285ffb640 3 client.4406.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:24:14.477 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:14.492+0000 7fa285ffb640 20 client.4406.objecter dump_active .. 0 homeless 2024-03-29T11:24:14.478 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:14.493+0000 7fa2954bf640 10 monclient: get_auth_request con 0x7fa27406dbc0 auth_method 0 2024-03-29T11:24:14.478 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:14.493+0000 7fa285ffb640 10 client.4406.objecter ms_handle_connect 0x7fa27406dbc0 2024-03-29T11:24:14.481 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:14.500+0000 7fa285ffb640 1 -- 172.21.15.84:0/3167863825 <== mon.2 v1:172.21.15.84:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fa2900cad30 con 0x7fa2880c8310 2024-03-29T11:24:14.481 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:14.500+0000 7fa285ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:24:14.481 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:14.500+0000 7fa285ffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:24:14.811 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:14.829+0000 7fa297723640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:24:14.812 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:14.829+0000 7fa297723640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:24:14.812 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:14.829+0000 7fa297723640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-29T11:24:14.812 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:14.829+0000 7fa297723640 1 -- 172.21.15.84:0/3167863825 --> v1:172.21.15.84:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa270005330 con 0x7fa2880c8310 2024-03-29T11:24:14.812 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:14.831+0000 7fa285ffb640 1 -- 172.21.15.84:0/3167863825 <== mon.2 v1:172.21.15.84:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4209477549) 0x7fa2900ccc60 con 0x7fa2880c8310 2024-03-29T11:24:14.812 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:14.831+0000 7fa285ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:24:14.812 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:14.831+0000 7fa285ffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:24:14.815 INFO:teuthology.misc.health.smithi084.stdout: 2024-03-29T11:24:14.815 INFO:teuthology.misc.health.smithi084.stdout:{"epoch":10,"fsid":"2c3e002f-9eb0-4860-853b-4ef1ef562213","created":"2024-03-29T11:19:32.029169+0000","modified":"2024-03-29T11:19:34.200299+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:34.200299+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"5c39e276-2034-4716-bbc0-57a634eefa92","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"cd580e99-109f-4bf1-88c0-a5987bc8b9ed","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"11e60a5a-051d-453f-a190-66387cb35592","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"2c329fbe-bbc4-4030-bf87-d711cad04679","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f4213d4a-bd36-4433-91ae-d7e2b02481d3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"f0cf44ea-6055-48af-805b-aed858102ff9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"d895eaa0-cc0b-4ee4-9b5a-2dfcd89a60b7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"9bc5a7bc-8f52-4db9-96d6-de5e68d52eaa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:24:14.817 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:14.836+0000 7fa297723640 10 librados: watch_flush enter 2024-03-29T11:24:14.817 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:14.836+0000 7fa297723640 10 librados: watch_flush exit 2024-03-29T11:24:14.817 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:14.836+0000 7fa297723640 20 client.4406.objecter shutdown clearing up homeless session... 2024-03-29T11:24:14.818 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:14.836+0000 7fa297723640 10 client.4406.objecter successfully canceled tick 2024-03-29T11:24:14.818 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:14.836+0000 7fa297723640 1 -- 172.21.15.84:0/3167863825 >> v1:172.21.15.179:6816/2981134980 conn(0x7fa27406dbc0 legacy=0x7fa274070080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:14.818 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:14.836+0000 7fa297723640 10 monclient: shutdown 2024-03-29T11:24:14.818 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:14.836+0000 7fa297723640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:24:14.818 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:14.836+0000 7fa297723640 1 -- 172.21.15.84:0/3167863825 >> v1:172.21.15.84:6790/0 conn(0x7fa2880c8310 legacy=0x7fa288167460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:14.818 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:14.837+0000 7fa297723640 1 -- 172.21.15.84:0/3167863825 shutdown_connections 2024-03-29T11:24:14.818 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:14.837+0000 7fa297723640 1 -- 172.21.15.84:0/3167863825 >> 172.21.15.84:0/3167863825 conn(0x7fa288005c00 msgr2=0x7fa288034090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:24:14.818 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:14.837+0000 7fa297723640 1 -- 172.21.15.84:0/3167863825 shutdown_connections 2024-03-29T11:24:14.818 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:14.837+0000 7fa297723640 1 -- 172.21.15.84:0/3167863825 wait complete. 2024-03-29T11:24:14.819 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:14.837+0000 7fa297723640 1 librados: shutdown 2024-03-29T11:24:14.837 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:24:21.644 DEBUG:teuthology.orchestra.run.smithi084:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:24:21.793 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:21.811+0000 7f61fd120640 10 monclient: get_monmap_and_config 2024-03-29T11:24:21.794 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:21.811+0000 7f61fd120640 10 monclient: build_initial_monmap 2024-03-29T11:24:21.794 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:21.811+0000 7f61fd120640 10 monclient: monmap: 2024-03-29T11:24:21.794 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-29T11:24:21.794 INFO:teuthology.misc.health.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:24:21.794 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T11:24:21.812238+0000 2024-03-29T11:24:21.794 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T11:24:21.812238+0000 2024-03-29T11:24:21.794 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-29T11:24:21.794 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:24:21.794 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-29T11:24:21.794 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.noname-c 2024-03-29T11:24:21.794 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-b 2024-03-29T11:24:21.794 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:24:21.797 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:21.813+0000 7f61fd120640 1 Processor -- start 2024-03-29T11:24:21.797 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:21.813+0000 7f61fd120640 1 -- start start 2024-03-29T11:24:21.797 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:21.813+0000 7f61fd120640 10 monclient: init 2024-03-29T11:24:21.797 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:21.814+0000 7f61fd120640 10 monclient: _reopen_session rank -1 2024-03-29T11:24:21.797 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:21.814+0000 7f61fd120640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:24:21.797 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:21.814+0000 7f61fd120640 10 monclient(hunting): picked mon.noname-c con 0x7f61f80a9750 addr v1:172.21.15.179:6789/0 2024-03-29T11:24:21.797 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:21.814+0000 7f61fd120640 10 monclient(hunting): picked mon.noname-b con 0x7f61f80adca0 addr v1:172.21.15.84:6790/0 2024-03-29T11:24:21.797 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:21.814+0000 7f61fd120640 10 monclient(hunting): picked mon.noname-a con 0x7f61f8146a80 addr v1:172.21.15.84:6789/0 2024-03-29T11:24:21.798 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:21.814+0000 7f61fd120640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f61f80abdc0 con 0x7f61f8146a80 2024-03-29T11:24:21.798 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:21.814+0000 7f61fd120640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f61f80b0070 con 0x7f61f80a9750 2024-03-29T11:24:21.798 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:21.814+0000 7f61fd120640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f61f80af880 con 0x7f61f80adca0 2024-03-29T11:24:21.798 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:21.814+0000 7f61fd120640 10 monclient(hunting): _renew_subs 2024-03-29T11:24:21.798 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:21.814+0000 7f61fd120640 10 monclient(hunting): authenticate will time out at 1123.108741s 2024-03-29T11:24:21.799 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:21.818+0000 7f61fc91f640 1 --1- >> v1:172.21.15.84:6789/0 conn(0x7f61f8146a80 0x7f61f8151bf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.84:6789/0 says I am v1:172.21.15.84:56175/0 (socket says 172.21.15.84:56175) 2024-03-29T11:24:21.800 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:21.818+0000 7f61fc91f640 1 -- 172.21.15.84:0/2290786408 learned_addr learned my addr 172.21.15.84:0/2290786408 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-29T11:24:21.800 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:21.818+0000 7f61f6ffd640 1 -- 172.21.15.84:0/2290786408 <== mon.0 v1:172.21.15.84:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2416237694 0 0) 0x7f61f80abdc0 con 0x7f61f8146a80 2024-03-29T11:24:21.800 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:21.818+0000 7f61f6ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:24:21.800 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:21.818+0000 7f61f6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:21.800 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:21.818+0000 7f61f6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:21.800 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:21.818+0000 7f61f6ffd640 10 monclient(hunting): my global_id is 4419 2024-03-29T11:24:21.800 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:21.818+0000 7f61f6ffd640 1 -- 172.21.15.84:0/2290786408 --> v1:172.21.15.84:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f61e0003520 con 0x7f61f8146a80 2024-03-29T11:24:21.801 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:21.819+0000 7f61f6ffd640 1 -- 172.21.15.84:0/2290786408 <== mon.1 v1:172.21.15.179:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3993203575 0 0) 0x7f61f80b0070 con 0x7f61f80a9750 2024-03-29T11:24:21.801 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:21.819+0000 7f61f6ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:24:21.801 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:21.819+0000 7f61f6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:21.801 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:21.819+0000 7f61f6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:21.802 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:21.819+0000 7f61f6ffd640 10 monclient(hunting): my global_id is 4372 2024-03-29T11:24:21.802 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:21.819+0000 7f61f6ffd640 1 -- 172.21.15.84:0/2290786408 --> v1:172.21.15.179:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f61f80abdc0 con 0x7f61f80a9750 2024-03-29T11:24:21.802 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:21.819+0000 7f61f6ffd640 1 -- 172.21.15.84:0/2290786408 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1857526337 0 0) 0x7f61f80af880 con 0x7f61f80adca0 2024-03-29T11:24:21.802 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:21.819+0000 7f61f6ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:24:21.802 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:21.819+0000 7f61f6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:21.802 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:21.819+0000 7f61f6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:21.802 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:21.819+0000 7f61f6ffd640 10 monclient(hunting): my global_id is 4409 2024-03-29T11:24:21.802 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:21.819+0000 7f61f6ffd640 1 -- 172.21.15.84:0/2290786408 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f61f80b0070 con 0x7f61f80adca0 2024-03-29T11:24:21.802 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:21.819+0000 7f61f6ffd640 1 -- 172.21.15.84:0/2290786408 <== mon.1 v1:172.21.15.179:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1503202112 0 0) 0x7f61f80abdc0 con 0x7f61f80a9750 2024-03-29T11:24:21.802 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:21.819+0000 7f61f6ffd640 1 -- 172.21.15.84:0/2290786408 >> v1:172.21.15.84:6790/0 conn(0x7f61f80adca0 legacy=0x7f61f8146370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:21.802 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:21.819+0000 7f61f6ffd640 1 -- 172.21.15.84:0/2290786408 >> v1:172.21.15.84:6789/0 conn(0x7f61f8146a80 legacy=0x7f61f8151bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:21.802 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:21.819+0000 7f61f6ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:24:21.802 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:21.819+0000 7f61f6ffd640 1 monclient: found mon.noname-c 2024-03-29T11:24:21.802 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:21.819+0000 7f61f6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:24:21.802 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:21.819+0000 7f61f6ffd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.179:6789/0 2024-03-29T11:24:21.803 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:21.819+0000 7f61f6ffd640 1 -- 172.21.15.84:0/2290786408 --> v1:172.21.15.179:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61f80ac030 con 0x7f61f80a9750 2024-03-29T11:24:21.803 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:21.819+0000 7f61f6ffd640 10 monclient: _finish_auth 0 2024-03-29T11:24:21.803 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:21.819+0000 7f61f6ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:24:21.803 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:21.819+0000 7f61f6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:24:21.803 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:21.819+0000 7f61f6ffd640 1 -- 172.21.15.84:0/2290786408 <== mon.1 v1:172.21.15.179:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4236868694 0 0) 0x7f61ec001670 con 0x7f61f80a9750 2024-03-29T11:24:21.803 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:21.819+0000 7f61f6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:21.803 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:21.819+0000 7f61f6ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:24:21.803 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:21.819+0000 7f61f6ffd640 10 monclient: dump: 2024-03-29T11:24:21.803 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T11:24:21.803 INFO:teuthology.misc.health.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:24:21.803 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T11:18:53.864025+0000 2024-03-29T11:24:21.803 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T11:18:53.864025+0000 2024-03-29T11:24:21.803 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:21.803 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:24:21.804 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T11:24:21.804 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T11:24:21.804 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T11:24:21.804 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:24:21.804 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:21.819+0000 7f61fd120640 5 monclient: authenticate success, global_id 4372 2024-03-29T11:24:21.804 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:21.819+0000 7f61fd120640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:24:21.804 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:21.820+0000 7f61f6ffd640 1 -- 172.21.15.84:0/2290786408 <== mon.1 v1:172.21.15.179:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f61ec0017d0 con 0x7f61f80a9750 2024-03-29T11:24:21.804 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:21.820+0000 7f61f6ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:24:21.804 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:21.820+0000 7f61f6ffd640 1 -- 172.21.15.84:0/2290786408 <== mon.1 v1:172.21.15.179:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4236868694 0 0) 0x7f61ec001c20 con 0x7f61f80a9750 2024-03-29T11:24:21.804 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:21.820+0000 7f61f6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:21.804 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:21.820+0000 7f61f6ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:24:21.804 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:21.820+0000 7f61f6ffd640 10 monclient: dump: 2024-03-29T11:24:21.804 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T11:24:21.804 INFO:teuthology.misc.health.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:24:21.804 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T11:18:53.864025+0000 2024-03-29T11:24:21.805 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T11:18:53.864025+0000 2024-03-29T11:24:21.805 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:21.805 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:24:21.805 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T11:24:21.805 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T11:24:21.805 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T11:24:21.805 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:24:21.805 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:21.821+0000 7f61fd120640 10 monclient: get_monmap_and_config success 2024-03-29T11:24:21.805 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:21.821+0000 7f61fd120640 4 set_mon_vals no callback set 2024-03-29T11:24:21.806 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:21.822+0000 7f61fd120640 10 monclient: shutdown 2024-03-29T11:24:21.806 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:21.822+0000 7f61fd120640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:24:21.806 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:21.822+0000 7f61fd120640 1 -- 172.21.15.84:0/2290786408 >> v1:172.21.15.179:6789/0 conn(0x7f61f80a9750 legacy=0x7f61f8143710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:21.806 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:21.823+0000 7f61fd120640 1 -- 172.21.15.84:0/2290786408 shutdown_connections 2024-03-29T11:24:21.806 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:21.823+0000 7f61fd120640 1 -- 172.21.15.84:0/2290786408 >> 172.21.15.84:0/2290786408 conn(0x7f61f80673e0 msgr2=0x7f61f80b3820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:24:21.807 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:21.826+0000 7f61fd120640 1 -- 172.21.15.84:0/2290786408 shutdown_connections 2024-03-29T11:24:21.807 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:21.826+0000 7f61fd120640 1 -- 172.21.15.84:0/2290786408 wait complete. 2024-03-29T11:24:21.809 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:21.827+0000 7f61fd120640 10 monclient: build_initial_monmap 2024-03-29T11:24:21.809 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:21.827+0000 7f61fd120640 10 monclient: monmap: 2024-03-29T11:24:21.809 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-29T11:24:21.809 INFO:teuthology.misc.health.smithi084.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:24:21.809 INFO:teuthology.misc.health.smithi084.stderr:last_changed 0.000000 2024-03-29T11:24:21.809 INFO:teuthology.misc.health.smithi084.stderr:created 0.000000 2024-03-29T11:24:21.809 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-29T11:24:21.809 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:24:21.809 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-29T11:24:21.809 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.noname-b 2024-03-29T11:24:21.809 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-c 2024-03-29T11:24:21.809 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:24:21.809 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:21.827+0000 7f61fd120640 1 librados: starting msgr at 2024-03-29T11:24:21.810 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:21.827+0000 7f61fd120640 1 librados: starting objecter 2024-03-29T11:24:21.810 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:21.827+0000 7f61fd120640 1 Processor -- start 2024-03-29T11:24:21.811 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:21.828+0000 7f61fd120640 1 -- start start 2024-03-29T11:24:21.812 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:21.828+0000 7f61fd120640 1 librados: setting wanted keys 2024-03-29T11:24:21.812 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:21.828+0000 7f61fd120640 1 librados: calling monclient init 2024-03-29T11:24:21.812 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:21.828+0000 7f61fd120640 10 monclient: init 2024-03-29T11:24:21.812 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:21.829+0000 7f61fd120640 10 monclient: _reopen_session rank -1 2024-03-29T11:24:21.812 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:21.829+0000 7f61fd120640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:24:21.812 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:21.829+0000 7f61fd120640 10 monclient(hunting): picked mon.noname-a con 0x7f61f80a9750 addr v1:172.21.15.84:6789/0 2024-03-29T11:24:21.812 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:21.829+0000 7f61fd120640 10 monclient(hunting): picked mon.noname-b con 0x7f61f80adca0 addr v1:172.21.15.179:6789/0 2024-03-29T11:24:21.812 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:21.829+0000 7f61fd120640 10 monclient(hunting): picked mon.noname-c con 0x7f61f8146a80 addr v1:172.21.15.84:6790/0 2024-03-29T11:24:21.812 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:21.829+0000 7f61fd120640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f61f813ebd0 con 0x7f61f80a9750 2024-03-29T11:24:21.812 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:21.829+0000 7f61fd120640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f61f80b33c0 con 0x7f61f80adca0 2024-03-29T11:24:21.812 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:21.829+0000 7f61fd120640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f61f81db160 con 0x7f61f8146a80 2024-03-29T11:24:21.812 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:21.829+0000 7f61fd120640 10 monclient(hunting): _renew_subs 2024-03-29T11:24:21.812 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:21.829+0000 7f61fd120640 10 monclient(hunting): authenticate will time out at 1123.123626s 2024-03-29T11:24:21.813 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:21.830+0000 7f61fc91f640 1 --1- >> v1:172.21.15.84:6790/0 conn(0x7f61f8146a80 0x7f61f813e4c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.84:6790/0 says I am v1:172.21.15.84:43472/0 (socket says 172.21.15.84:43472) 2024-03-29T11:24:21.813 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:21.830+0000 7f61fc91f640 1 -- 172.21.15.84:0/3404157364 learned_addr learned my addr 172.21.15.84:0/3404157364 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-29T11:24:21.813 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:21.830+0000 7f61f4ff9640 10 client.?.objecter ms_handle_connect 0x7f61f8146a80 2024-03-29T11:24:21.813 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:21.830+0000 7f61f4ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:24:21.815 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:21.831+0000 7f61f4ff9640 1 -- 172.21.15.84:0/3404157364 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1170677849 0 0) 0x7f61f81db160 con 0x7f61f8146a80 2024-03-29T11:24:21.815 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:21.831+0000 7f61f4ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:24:21.815 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:21.831+0000 7f61f4ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:21.815 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:21.831+0000 7f61f4ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:21.815 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:21.831+0000 7f61f4ff9640 10 monclient(hunting): my global_id is 4412 2024-03-29T11:24:21.815 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:21.831+0000 7f61f4ff9640 1 -- 172.21.15.84:0/3404157364 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f61e4004eb0 con 0x7f61f8146a80 2024-03-29T11:24:21.815 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:21.831+0000 7f61f4ff9640 1 -- 172.21.15.84:0/3404157364 <== mon.2 v1:172.21.15.84:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4228106667 0 0) 0x7f61e4004eb0 con 0x7f61f8146a80 2024-03-29T11:24:21.815 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:21.832+0000 7f61f4ff9640 1 -- 172.21.15.84:0/3404157364 --> v1:172.21.15.84:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f61f81db160 con 0x7f61f8146a80 2024-03-29T11:24:21.815 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:21.832+0000 7f61f4ff9640 1 -- 172.21.15.84:0/3404157364 <== mon.2 v1:172.21.15.84:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4236868694 0 0) 0x7f61f0001b50 con 0x7f61f8146a80 2024-03-29T11:24:21.815 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:21.832+0000 7f61f4ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:21.816 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:21.832+0000 7f61f4ff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:24:21.816 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:21.832+0000 7f61f4ff9640 10 monclient(hunting): dump: 2024-03-29T11:24:21.816 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T11:24:21.816 INFO:teuthology.misc.health.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:24:21.816 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T11:18:53.864025+0000 2024-03-29T11:24:21.816 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T11:18:53.864025+0000 2024-03-29T11:24:21.816 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:21.816 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:24:21.816 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T11:24:21.816 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T11:24:21.816 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T11:24:21.816 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:24:21.816 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:21.832+0000 7f61f4ff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:24:21.816 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:21.832+0000 7f61f4ff9640 1 -- 172.21.15.84:0/3404157364 <== mon.2 v1:172.21.15.84:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3508171525 0 0) 0x7f61f81db160 con 0x7f61f8146a80 2024-03-29T11:24:21.817 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:21.832+0000 7f61f4ff9640 1 -- 172.21.15.84:0/3404157364 >> v1:172.21.15.179:6789/0 conn(0x7f61f80adca0 legacy=0x7f61f813ddb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:24:21.817 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:21.833+0000 7f61f4ff9640 1 -- 172.21.15.84:0/3404157364 >> v1:172.21.15.84:6789/0 conn(0x7f61f80a9750 legacy=0x7f61f81db680 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:24:21.817 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:21.833+0000 7f61f4ff9640 10 monclient: _finish_hunting 0 2024-03-29T11:24:21.817 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:21.833+0000 7f61f4ff9640 1 monclient: found mon.c 2024-03-29T11:24:21.817 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:21.833+0000 7f61f4ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:24:21.817 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:21.833+0000 7f61f4ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-29T11:24:21.817 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:21.833+0000 7f61f4ff9640 1 -- 172.21.15.84:0/3404157364 --> v1:172.21.15.84:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61f81dbd90 con 0x7f61f8146a80 2024-03-29T11:24:21.817 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:21.833+0000 7f61f4ff9640 10 monclient: _finish_auth 0 2024-03-29T11:24:21.817 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:21.833+0000 7f61f4ff9640 10 monclient: _check_auth_tickets 2024-03-29T11:24:21.817 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:21.833+0000 7f61f4ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:24:21.819 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:21.835+0000 7f61fd120640 5 monclient: authenticate success, global_id 4412 2024-03-29T11:24:21.819 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:21.835+0000 7f61fd120640 10 monclient: _renew_subs 2024-03-29T11:24:21.819 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:21.835+0000 7f61fd120640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-29T11:24:21.819 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:21.835+0000 7f61fd120640 1 -- 172.21.15.84:0/3404157364 --> v1:172.21.15.84:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f61f81460e0 con 0x7f61f8146a80 2024-03-29T11:24:21.819 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:21.835+0000 7f61fd120640 10 client.4412.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:24:21.819 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:21.835+0000 7f61fd120640 10 monclient: _renew_subs 2024-03-29T11:24:21.819 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:21.835+0000 7f61fd120640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-29T11:24:21.819 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:21.835+0000 7f61fd120640 1 -- 172.21.15.84:0/3404157364 --> v1:172.21.15.84:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f61f8144350 con 0x7f61f8146a80 2024-03-29T11:24:21.819 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:21.835+0000 7f61fd120640 1 librados: init done 2024-03-29T11:24:21.819 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:21.836+0000 7f61f4ff9640 1 -- 172.21.15.84:0/3404157364 <== mon.2 v1:172.21.15.84:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f61f0001cb0 con 0x7f61f8146a80 2024-03-29T11:24:21.819 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:21.836+0000 7f61f4ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:24:21.819 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:21.836+0000 7f61f4ff9640 1 -- 172.21.15.84:0/3404157364 <== mon.2 v1:172.21.15.84:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4236868694 0 0) 0x7f61f0004dc0 con 0x7f61f8146a80 2024-03-29T11:24:21.819 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:21.836+0000 7f61f4ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:21.819 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:21.836+0000 7f61f4ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:24:21.820 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:21.836+0000 7f61f4ff9640 10 monclient: dump: 2024-03-29T11:24:21.820 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T11:24:21.820 INFO:teuthology.misc.health.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:24:21.820 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T11:18:53.864025+0000 2024-03-29T11:24:21.820 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T11:18:53.864025+0000 2024-03-29T11:24:21.820 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:21.820 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:24:21.820 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T11:24:21.820 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T11:24:21.820 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T11:24:21.820 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:24:21.820 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:21.837+0000 7f61fd120640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:24:21.820 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:21.837+0000 7f61fd120640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:24:21.820 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:21.837+0000 7f61fd120640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-29T11:24:21.820 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:21.837+0000 7f61fd120640 1 -- 172.21.15.84:0/3404157364 --> v1:172.21.15.84:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f61f81db350 con 0x7f61f8146a80 2024-03-29T11:24:21.821 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:21.837+0000 7f61f67fc640 4 set_mon_vals no callback set 2024-03-29T11:24:21.821 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:21.840+0000 7f61f4ff9640 1 -- 172.21.15.84:0/3404157364 <== mon.2 v1:172.21.15.84:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4033942909 0 0) 0x7f61f0005000 con 0x7f61f8146a80 2024-03-29T11:24:21.822 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:21.841+0000 7f61f7fff640 10 monclient: get_auth_request con 0x7f61e406dc80 auth_method 0 2024-03-29T11:24:21.822 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:21.841+0000 7f61f4ff9640 1 -- 172.21.15.84:0/3404157364 <== mon.2 v1:172.21.15.84:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2808139587 0 0) 0x7f61f0084d70 con 0x7f61f8146a80 2024-03-29T11:24:21.823 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:21.841+0000 7f61f4ff9640 10 client.4412.objecter ms_dispatch 0x7f61f8000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:24:21.823 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:21.841+0000 7f61f4ff9640 3 client.4412.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:24:21.823 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:21.841+0000 7f61f4ff9640 3 client.4412.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:24:21.823 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:21.841+0000 7f61f4ff9640 20 client.4412.objecter dump_active .. 0 homeless 2024-03-29T11:24:21.823 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:21.841+0000 7f61f4ff9640 10 client.4412.objecter ms_handle_connect 0x7f61e406dc80 2024-03-29T11:24:21.827 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:21.846+0000 7f61f4ff9640 1 -- 172.21.15.84:0/3404157364 <== mon.2 v1:172.21.15.84:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f61f0050650 con 0x7f61f8146a80 2024-03-29T11:24:21.827 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:21.846+0000 7f61f4ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:24:21.827 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:21.846+0000 7f61f4ff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:24:22.169 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:22.187+0000 7f61fd120640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:24:22.169 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:22.187+0000 7f61fd120640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:24:22.169 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:22.187+0000 7f61fd120640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-29T11:24:22.169 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:22.187+0000 7f61fd120640 1 -- 172.21.15.84:0/3404157364 --> v1:172.21.15.84:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f61f804f730 con 0x7f61f8146a80 2024-03-29T11:24:22.169 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:22.188+0000 7f61f4ff9640 1 -- 172.21.15.84:0/3404157364 <== mon.2 v1:172.21.15.84:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4209477549) 0x7f61f0052580 con 0x7f61f8146a80 2024-03-29T11:24:22.169 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:22.188+0000 7f61f4ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:24:22.170 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:22.188+0000 7f61f4ff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:24:22.173 INFO:teuthology.misc.health.smithi084.stdout: 2024-03-29T11:24:22.173 INFO:teuthology.misc.health.smithi084.stdout:{"epoch":10,"fsid":"2c3e002f-9eb0-4860-853b-4ef1ef562213","created":"2024-03-29T11:19:32.029169+0000","modified":"2024-03-29T11:19:34.200299+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:34.200299+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"5c39e276-2034-4716-bbc0-57a634eefa92","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"cd580e99-109f-4bf1-88c0-a5987bc8b9ed","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"11e60a5a-051d-453f-a190-66387cb35592","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"2c329fbe-bbc4-4030-bf87-d711cad04679","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f4213d4a-bd36-4433-91ae-d7e2b02481d3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"f0cf44ea-6055-48af-805b-aed858102ff9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"d895eaa0-cc0b-4ee4-9b5a-2dfcd89a60b7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"9bc5a7bc-8f52-4db9-96d6-de5e68d52eaa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:24:22.174 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:22.192+0000 7f61fd120640 10 librados: watch_flush enter 2024-03-29T11:24:22.174 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:22.192+0000 7f61fd120640 10 librados: watch_flush exit 2024-03-29T11:24:22.174 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:22.192+0000 7f61fd120640 20 client.4412.objecter shutdown clearing up homeless session... 2024-03-29T11:24:22.174 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:22.192+0000 7f61fd120640 10 client.4412.objecter successfully canceled tick 2024-03-29T11:24:22.174 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:22.192+0000 7f61fd120640 1 -- 172.21.15.84:0/3404157364 >> v1:172.21.15.179:6816/2981134980 conn(0x7f61e406dc80 legacy=0x7f61e4070140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:22.174 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:22.192+0000 7f61fd120640 10 monclient: shutdown 2024-03-29T11:24:22.174 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:22.192+0000 7f61fd120640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:24:22.174 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:22.192+0000 7f61fd120640 1 -- 172.21.15.84:0/3404157364 >> v1:172.21.15.84:6790/0 conn(0x7f61f8146a80 legacy=0x7f61f813e4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:22.174 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:22.192+0000 7f61fd120640 1 -- 172.21.15.84:0/3404157364 shutdown_connections 2024-03-29T11:24:22.174 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:22.192+0000 7f61fd120640 1 -- 172.21.15.84:0/3404157364 >> 172.21.15.84:0/3404157364 conn(0x7f61f80673e0 msgr2=0x7f61f80aa200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:24:22.175 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:22.193+0000 7f61fd120640 1 -- 172.21.15.84:0/3404157364 shutdown_connections 2024-03-29T11:24:22.175 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:22.193+0000 7f61fd120640 1 -- 172.21.15.84:0/3404157364 wait complete. 2024-03-29T11:24:22.175 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:22.193+0000 7f61fd120640 1 librados: shutdown 2024-03-29T11:24:22.193 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:24:28.999 DEBUG:teuthology.orchestra.run.smithi084:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:24:29.146 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:29.162+0000 7f08663ba640 10 monclient: get_monmap_and_config 2024-03-29T11:24:29.149 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:29.162+0000 7f08663ba640 10 monclient: build_initial_monmap 2024-03-29T11:24:29.149 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:29.162+0000 7f08663ba640 10 monclient: monmap: 2024-03-29T11:24:29.149 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-29T11:24:29.149 INFO:teuthology.misc.health.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:24:29.149 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T11:24:29.164476+0000 2024-03-29T11:24:29.149 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T11:24:29.164476+0000 2024-03-29T11:24:29.149 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-29T11:24:29.150 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:24:29.150 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-29T11:24:29.150 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.noname-c 2024-03-29T11:24:29.150 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-b 2024-03-29T11:24:29.150 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:24:29.150 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:29.165+0000 7f08663ba640 1 Processor -- start 2024-03-29T11:24:29.151 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:29.165+0000 7f08663ba640 1 -- start start 2024-03-29T11:24:29.151 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:29.165+0000 7f08663ba640 10 monclient: init 2024-03-29T11:24:29.151 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:29.166+0000 7f08663ba640 10 monclient: _reopen_session rank -1 2024-03-29T11:24:29.151 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:29.166+0000 7f08663ba640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:24:29.151 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:29.166+0000 7f08663ba640 10 monclient(hunting): picked mon.noname-c con 0x7f08600a9ac0 addr v1:172.21.15.179:6789/0 2024-03-29T11:24:29.151 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:29.166+0000 7f08663ba640 10 monclient(hunting): picked mon.noname-a con 0x7f086013ee90 addr v1:172.21.15.84:6789/0 2024-03-29T11:24:29.151 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:29.166+0000 7f08663ba640 10 monclient(hunting): picked mon.noname-b con 0x7f08600af900 addr v1:172.21.15.84:6790/0 2024-03-29T11:24:29.151 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:29.166+0000 7f08663ba640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f08600afd10 con 0x7f086013ee90 2024-03-29T11:24:29.151 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:29.166+0000 7f08663ba640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f08600af6f0 con 0x7f08600a9ac0 2024-03-29T11:24:29.151 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:29.166+0000 7f08663ba640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f08600aac80 con 0x7f08600af900 2024-03-29T11:24:29.151 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:29.166+0000 7f08663ba640 10 monclient(hunting): _renew_subs 2024-03-29T11:24:29.151 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:29.166+0000 7f08663ba640 10 monclient(hunting): authenticate will time out at 1130.461659s 2024-03-29T11:24:29.159 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:29.172+0000 7f0864957640 1 --1- >> v1:172.21.15.84:6790/0 conn(0x7f08600af900 0x7f0860149b00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.84:6790/0 says I am v1:172.21.15.84:46611/0 (socket says 172.21.15.84:46611) 2024-03-29T11:24:29.159 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:29.172+0000 7f0864957640 1 -- 172.21.15.84:0/3829574458 learned_addr learned my addr 172.21.15.84:0/3829574458 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-29T11:24:29.159 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:29.172+0000 7f085f7fe640 1 --1- >> v1:172.21.15.84:6789/0 conn(0x7f086013ee90 0x7f086013b5f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.84:6789/0 says I am v1:172.21.15.84:35453/0 (socket says 172.21.15.84:35453) 2024-03-29T11:24:29.159 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:29.172+0000 7f085effd640 1 -- 172.21.15.84:0/3829574458 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 353237027 0 0) 0x7f08600aac80 con 0x7f08600af900 2024-03-29T11:24:29.159 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:29.172+0000 7f085effd640 10 monclient(hunting): _negotiate 2024-03-29T11:24:29.159 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:29.172+0000 7f085effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:29.159 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:29.172+0000 7f085effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:29.159 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:29.172+0000 7f085effd640 10 monclient(hunting): my global_id is 4415 2024-03-29T11:24:29.159 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:29.172+0000 7f085effd640 1 -- 172.21.15.84:0/3829574458 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f084c003520 con 0x7f08600af900 2024-03-29T11:24:29.160 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:29.172+0000 7f085effd640 1 -- 172.21.15.84:0/3829574458 <== mon.0 v1:172.21.15.84:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1793663126 0 0) 0x7f08600afd10 con 0x7f086013ee90 2024-03-29T11:24:29.160 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:29.172+0000 7f085effd640 10 monclient(hunting): _negotiate 2024-03-29T11:24:29.160 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:29.172+0000 7f085effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:29.160 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:29.172+0000 7f085effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:29.160 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:29.172+0000 7f085effd640 10 monclient(hunting): my global_id is 4422 2024-03-29T11:24:29.160 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:29.172+0000 7f085effd640 1 -- 172.21.15.84:0/3829574458 --> v1:172.21.15.84:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f08600aac80 con 0x7f086013ee90 2024-03-29T11:24:29.160 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:29.172+0000 7f085effd640 1 -- 172.21.15.84:0/3829574458 <== mon.1 v1:172.21.15.179:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 545123498 0 0) 0x7f08600af6f0 con 0x7f08600a9ac0 2024-03-29T11:24:29.160 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:29.172+0000 7f085effd640 10 monclient(hunting): _negotiate 2024-03-29T11:24:29.160 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:29.172+0000 7f085effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:29.160 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:29.172+0000 7f085effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:29.160 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:29.172+0000 7f085effd640 10 monclient(hunting): my global_id is 4375 2024-03-29T11:24:29.160 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:29.172+0000 7f085effd640 1 -- 172.21.15.84:0/3829574458 --> v1:172.21.15.179:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f08600afd10 con 0x7f08600a9ac0 2024-03-29T11:24:29.160 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:29.173+0000 7f085effd640 1 -- 172.21.15.84:0/3829574458 <== mon.1 v1:172.21.15.179:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3933494996 0 0) 0x7f08600afd10 con 0x7f08600a9ac0 2024-03-29T11:24:29.160 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:29.173+0000 7f085effd640 1 -- 172.21.15.84:0/3829574458 >> v1:172.21.15.84:6790/0 conn(0x7f08600af900 legacy=0x7f0860149b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:29.161 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:29.173+0000 7f085effd640 1 -- 172.21.15.84:0/3829574458 >> v1:172.21.15.84:6789/0 conn(0x7f086013ee90 legacy=0x7f086013b5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:29.161 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:29.173+0000 7f085effd640 10 monclient: _finish_hunting 0 2024-03-29T11:24:29.161 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:29.173+0000 7f085effd640 1 monclient: found mon.noname-c 2024-03-29T11:24:29.161 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:29.173+0000 7f085effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:24:29.161 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:29.173+0000 7f085effd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.179:6789/0 2024-03-29T11:24:29.161 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:29.173+0000 7f085effd640 1 -- 172.21.15.84:0/3829574458 --> v1:172.21.15.179:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08600aaf60 con 0x7f08600a9ac0 2024-03-29T11:24:29.161 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:29.173+0000 7f085effd640 10 monclient: _finish_auth 0 2024-03-29T11:24:29.161 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:29.173+0000 7f085effd640 10 monclient: _check_auth_tickets 2024-03-29T11:24:29.161 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:29.173+0000 7f085effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:24:29.161 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:29.174+0000 7f08663ba640 5 monclient: authenticate success, global_id 4375 2024-03-29T11:24:29.161 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:29.174+0000 7f08663ba640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:24:29.161 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:29.174+0000 7f085effd640 1 -- 172.21.15.84:0/3829574458 <== mon.1 v1:172.21.15.179:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4236868694 0 0) 0x7f08580014e0 con 0x7f08600a9ac0 2024-03-29T11:24:29.161 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:29.174+0000 7f085effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:29.161 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:29.174+0000 7f085effd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:24:29.161 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:29.174+0000 7f085effd640 10 monclient: dump: 2024-03-29T11:24:29.162 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T11:24:29.162 INFO:teuthology.misc.health.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:24:29.162 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T11:18:53.864025+0000 2024-03-29T11:24:29.162 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T11:18:53.864025+0000 2024-03-29T11:24:29.162 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:29.162 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:24:29.162 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T11:24:29.162 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T11:24:29.162 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T11:24:29.162 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:24:29.162 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:29.174+0000 7f08663ba640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:24:29.162 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:29.174+0000 7f085effd640 1 -- 172.21.15.84:0/3829574458 <== mon.1 v1:172.21.15.179:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0858001700 con 0x7f08600a9ac0 2024-03-29T11:24:29.162 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:29.174+0000 7f085effd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:24:29.162 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:29.174+0000 7f08663ba640 10 monclient: get_monmap_and_config success 2024-03-29T11:24:29.163 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:29.174+0000 7f08663ba640 4 set_mon_vals no callback set 2024-03-29T11:24:29.163 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:29.175+0000 7f08663ba640 10 monclient: shutdown 2024-03-29T11:24:29.163 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:29.175+0000 7f08663ba640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:24:29.163 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:29.175+0000 7f08663ba640 1 -- 172.21.15.84:0/3829574458 >> v1:172.21.15.179:6789/0 conn(0x7f08600a9ac0 legacy=0x7f08600adfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:29.163 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:29.175+0000 7f08663ba640 1 -- 172.21.15.84:0/3829574458 shutdown_connections 2024-03-29T11:24:29.163 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:29.175+0000 7f08663ba640 1 -- 172.21.15.84:0/3829574458 >> 172.21.15.84:0/3829574458 conn(0x7f08600b35e0 msgr2=0x7f08600b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:24:29.163 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:29.175+0000 7f08663ba640 1 -- 172.21.15.84:0/3829574458 shutdown_connections 2024-03-29T11:24:29.163 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:29.175+0000 7f08663ba640 1 -- 172.21.15.84:0/3829574458 wait complete. 2024-03-29T11:24:29.164 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:29.176+0000 7f08663ba640 10 monclient: build_initial_monmap 2024-03-29T11:24:29.164 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:29.176+0000 7f08663ba640 10 monclient: monmap: 2024-03-29T11:24:29.164 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-29T11:24:29.164 INFO:teuthology.misc.health.smithi084.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:24:29.164 INFO:teuthology.misc.health.smithi084.stderr:last_changed 0.000000 2024-03-29T11:24:29.164 INFO:teuthology.misc.health.smithi084.stderr:created 0.000000 2024-03-29T11:24:29.164 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-29T11:24:29.164 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:24:29.164 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-29T11:24:29.165 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.noname-b 2024-03-29T11:24:29.165 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-c 2024-03-29T11:24:29.165 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:24:29.165 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:29.177+0000 7f08663ba640 1 librados: starting msgr at 2024-03-29T11:24:29.165 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:29.177+0000 7f08663ba640 1 librados: starting objecter 2024-03-29T11:24:29.165 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:29.177+0000 7f08663ba640 1 Processor -- start 2024-03-29T11:24:29.165 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:29.177+0000 7f08663ba640 1 -- start start 2024-03-29T11:24:29.165 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:29.177+0000 7f08663ba640 1 librados: setting wanted keys 2024-03-29T11:24:29.165 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:29.177+0000 7f08663ba640 1 librados: calling monclient init 2024-03-29T11:24:29.165 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:29.177+0000 7f08663ba640 10 monclient: init 2024-03-29T11:24:29.165 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:29.178+0000 7f08663ba640 10 monclient: _reopen_session rank -1 2024-03-29T11:24:29.165 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:29.178+0000 7f08663ba640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:24:29.165 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:29.178+0000 7f08663ba640 10 monclient(hunting): picked mon.noname-a con 0x7f08600af900 addr v1:172.21.15.84:6789/0 2024-03-29T11:24:29.165 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:29.178+0000 7f08663ba640 10 monclient(hunting): picked mon.noname-b con 0x7f086013ee90 addr v1:172.21.15.179:6789/0 2024-03-29T11:24:29.166 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:29.178+0000 7f08663ba640 10 monclient(hunting): picked mon.noname-c con 0x7f08601417c0 addr v1:172.21.15.84:6790/0 2024-03-29T11:24:29.166 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:29.178+0000 7f08663ba640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0860140680 con 0x7f08600af900 2024-03-29T11:24:29.166 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:29.178+0000 7f08663ba640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f086006e8d0 con 0x7f086013ee90 2024-03-29T11:24:29.166 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:29.178+0000 7f08663ba640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f086006ebb0 con 0x7f08601417c0 2024-03-29T11:24:29.166 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:29.178+0000 7f08663ba640 10 monclient(hunting): _renew_subs 2024-03-29T11:24:29.166 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:29.178+0000 7f08663ba640 10 monclient(hunting): authenticate will time out at 1130.473701s 2024-03-29T11:24:29.171 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:29.183+0000 7f085f7fe640 1 --1- >> v1:172.21.15.179:6789/0 conn(0x7f086013ee90 0x7f08601df440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.179:6789/0 says I am v1:172.21.15.84:41379/0 (socket says 172.21.15.84:41379) 2024-03-29T11:24:29.171 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:29.183+0000 7f085f7fe640 1 -- 172.21.15.84:0/3545294764 learned_addr learned my addr 172.21.15.84:0/3545294764 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-29T11:24:29.171 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:29.183+0000 7f085cff9640 10 client.?.objecter ms_handle_connect 0x7f086013ee90 2024-03-29T11:24:29.171 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:29.184+0000 7f085cff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:24:29.171 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:29.184+0000 7f085cff9640 1 -- 172.21.15.84:0/3545294764 <== mon.1 v1:172.21.15.179:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 692126691 0 0) 0x7f086006e8d0 con 0x7f086013ee90 2024-03-29T11:24:29.171 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:29.184+0000 7f085cff9640 10 monclient(hunting): _negotiate 2024-03-29T11:24:29.171 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:29.184+0000 7f085cff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:29.171 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:29.184+0000 7f085cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:29.171 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:29.184+0000 7f085cff9640 10 monclient(hunting): my global_id is 4378 2024-03-29T11:24:29.172 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:29.184+0000 7f085cff9640 1 -- 172.21.15.84:0/3545294764 --> v1:172.21.15.179:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0840004eb0 con 0x7f086013ee90 2024-03-29T11:24:29.172 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:29.184+0000 7f085cff9640 10 client.?.objecter ms_handle_connect 0x7f08601417c0 2024-03-29T11:24:29.172 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:29.184+0000 7f085cff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:24:29.172 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:29.184+0000 7f085cff9640 1 -- 172.21.15.84:0/3545294764 <== mon.1 v1:172.21.15.179:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4241882861 0 0) 0x7f0840004eb0 con 0x7f086013ee90 2024-03-29T11:24:29.172 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:29.185+0000 7f085cff9640 1 -- 172.21.15.84:0/3545294764 --> v1:172.21.15.179:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f086006e8d0 con 0x7f086013ee90 2024-03-29T11:24:29.172 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:29.185+0000 7f085cff9640 1 -- 172.21.15.84:0/3545294764 <== mon.1 v1:172.21.15.179:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4236868694 0 0) 0x7f0848001800 con 0x7f086013ee90 2024-03-29T11:24:29.172 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:29.185+0000 7f085cff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:29.172 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:29.185+0000 7f085cff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:24:29.172 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:29.185+0000 7f085cff9640 10 monclient(hunting): dump: 2024-03-29T11:24:29.172 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T11:24:29.172 INFO:teuthology.misc.health.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:24:29.172 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T11:18:53.864025+0000 2024-03-29T11:24:29.172 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T11:18:53.864025+0000 2024-03-29T11:24:29.172 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:29.172 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:24:29.173 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T11:24:29.173 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T11:24:29.173 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T11:24:29.173 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:24:29.173 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:29.185+0000 7f085cff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:24:29.173 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:29.185+0000 7f085cff9640 1 -- 172.21.15.84:0/3545294764 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3722419159 0 0) 0x7f086006ebb0 con 0x7f08601417c0 2024-03-29T11:24:29.173 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:29.185+0000 7f085cff9640 10 monclient(hunting): _negotiate 2024-03-29T11:24:29.173 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:29.185+0000 7f085cff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:29.173 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:29.185+0000 7f085cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:29.173 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:29.185+0000 7f085cff9640 10 monclient(hunting): my global_id is 4418 2024-03-29T11:24:29.173 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:29.185+0000 7f085cff9640 1 -- 172.21.15.84:0/3545294764 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0840004eb0 con 0x7f08601417c0 2024-03-29T11:24:29.173 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:29.185+0000 7f085cff9640 1 -- 172.21.15.84:0/3545294764 <== mon.1 v1:172.21.15.179:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3239285554 0 0) 0x7f086006e8d0 con 0x7f086013ee90 2024-03-29T11:24:29.173 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:29.185+0000 7f085cff9640 1 -- 172.21.15.84:0/3545294764 >> v1:172.21.15.84:6790/0 conn(0x7f08601417c0 legacy=0x7f08601e6290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:29.173 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:29.185+0000 7f085cff9640 1 -- 172.21.15.84:0/3545294764 >> v1:172.21.15.84:6789/0 conn(0x7f08600af900 legacy=0x7f08601ded30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:29.174 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:29.186+0000 7f085cff9640 10 monclient: _finish_hunting 0 2024-03-29T11:24:29.174 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:29.186+0000 7f085cff9640 1 monclient: found mon.b 2024-03-29T11:24:29.174 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:29.186+0000 7f085cff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:24:29.174 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:29.186+0000 7f085cff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-29T11:24:29.174 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:29.186+0000 7f085cff9640 1 -- 172.21.15.84:0/3545294764 --> v1:172.21.15.179:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f086013d480 con 0x7f086013ee90 2024-03-29T11:24:29.174 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:29.186+0000 7f085cff9640 10 monclient: _finish_auth 0 2024-03-29T11:24:29.174 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:29.186+0000 7f085cff9640 10 monclient: _check_auth_tickets 2024-03-29T11:24:29.174 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:29.186+0000 7f085cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:24:29.174 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:29.186+0000 7f08663ba640 5 monclient: authenticate success, global_id 4378 2024-03-29T11:24:29.174 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:29.186+0000 7f08663ba640 10 monclient: _renew_subs 2024-03-29T11:24:29.174 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:29.186+0000 7f08663ba640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-29T11:24:29.174 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:29.186+0000 7f08663ba640 1 -- 172.21.15.84:0/3545294764 --> v1:172.21.15.179:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f086013d6f0 con 0x7f086013ee90 2024-03-29T11:24:29.174 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:29.186+0000 7f08663ba640 10 client.4378.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:24:29.174 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:29.186+0000 7f08663ba640 10 monclient: _renew_subs 2024-03-29T11:24:29.174 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:29.186+0000 7f08663ba640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-29T11:24:29.175 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:29.186+0000 7f08663ba640 1 -- 172.21.15.84:0/3545294764 --> v1:172.21.15.179:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f08601e7b90 con 0x7f086013ee90 2024-03-29T11:24:29.175 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:29.186+0000 7f08663ba640 1 librados: init done 2024-03-29T11:24:29.175 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:29.186+0000 7f085cff9640 1 -- 172.21.15.84:0/3545294764 <== mon.1 v1:172.21.15.179:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0848001960 con 0x7f086013ee90 2024-03-29T11:24:29.175 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:29.186+0000 7f085cff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:24:29.175 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:29.187+0000 7f085cff9640 1 -- 172.21.15.84:0/3545294764 <== mon.1 v1:172.21.15.179:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4236868694 0 0) 0x7f0848004960 con 0x7f086013ee90 2024-03-29T11:24:29.175 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:29.187+0000 7f085e7fc640 4 set_mon_vals no callback set 2024-03-29T11:24:29.175 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:29.188+0000 7f08663ba640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:24:29.176 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:29.189+0000 7f08663ba640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:24:29.176 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:29.189+0000 7f08663ba640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-29T11:24:29.176 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:29.189+0000 7f08663ba640 1 -- 172.21.15.84:0/3545294764 --> v1:172.21.15.179:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0844005080 con 0x7f086013ee90 2024-03-29T11:24:29.176 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:29.189+0000 7f085cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:29.176 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:29.189+0000 7f085cff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:24:29.176 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:29.189+0000 7f085cff9640 10 monclient: dump: 2024-03-29T11:24:29.176 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T11:24:29.177 INFO:teuthology.misc.health.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:24:29.177 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T11:18:53.864025+0000 2024-03-29T11:24:29.177 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T11:18:53.864025+0000 2024-03-29T11:24:29.177 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:29.177 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:24:29.177 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T11:24:29.177 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T11:24:29.177 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T11:24:29.177 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:24:29.177 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:29.189+0000 7f085cff9640 1 -- 172.21.15.84:0/3545294764 <== mon.1 v1:172.21.15.179:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4033942909 0 0) 0x7f0848004ac0 con 0x7f086013ee90 2024-03-29T11:24:29.177 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:29.190+0000 7f085ffff640 10 monclient: get_auth_request con 0x7f084006ddb0 auth_method 0 2024-03-29T11:24:29.177 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:29.190+0000 7f085cff9640 1 -- 172.21.15.84:0/3545294764 <== mon.1 v1:172.21.15.179:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2808139587 0 0) 0x7f0848084a50 con 0x7f086013ee90 2024-03-29T11:24:29.177 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:29.190+0000 7f085cff9640 10 client.4378.objecter ms_dispatch 0x7f0860000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:24:29.178 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:29.190+0000 7f085cff9640 3 client.4378.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:24:29.178 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:29.190+0000 7f085cff9640 3 client.4378.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:24:29.178 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:29.190+0000 7f085cff9640 20 client.4378.objecter dump_active .. 0 homeless 2024-03-29T11:24:29.178 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:29.190+0000 7f085cff9640 10 client.4378.objecter ms_handle_connect 0x7f084006ddb0 2024-03-29T11:24:29.180 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:29.197+0000 7f085cff9640 1 -- 172.21.15.84:0/3545294764 <== mon.1 v1:172.21.15.179:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f0848050330 con 0x7f086013ee90 2024-03-29T11:24:29.180 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:29.197+0000 7f085cff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:24:29.180 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:29.197+0000 7f085cff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:24:29.506 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:29.523+0000 7f08663ba640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:24:29.507 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:29.524+0000 7f08663ba640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:24:29.507 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:29.524+0000 7f08663ba640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-29T11:24:29.507 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:29.524+0000 7f08663ba640 1 -- 172.21.15.84:0/3545294764 --> v1:172.21.15.179:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0844005330 con 0x7f086013ee90 2024-03-29T11:24:29.509 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:29.527+0000 7f085cff9640 1 -- 172.21.15.84:0/3545294764 <== mon.1 v1:172.21.15.179:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4209477549) 0x7f0848052260 con 0x7f086013ee90 2024-03-29T11:24:29.509 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:29.527+0000 7f085cff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:24:29.510 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:29.527+0000 7f085cff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:24:29.510 INFO:teuthology.misc.health.smithi084.stdout: 2024-03-29T11:24:29.510 INFO:teuthology.misc.health.smithi084.stdout:{"epoch":10,"fsid":"2c3e002f-9eb0-4860-853b-4ef1ef562213","created":"2024-03-29T11:19:32.029169+0000","modified":"2024-03-29T11:19:34.200299+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:34.200299+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"5c39e276-2034-4716-bbc0-57a634eefa92","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"cd580e99-109f-4bf1-88c0-a5987bc8b9ed","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"11e60a5a-051d-453f-a190-66387cb35592","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"2c329fbe-bbc4-4030-bf87-d711cad04679","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f4213d4a-bd36-4433-91ae-d7e2b02481d3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"f0cf44ea-6055-48af-805b-aed858102ff9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"d895eaa0-cc0b-4ee4-9b5a-2dfcd89a60b7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"9bc5a7bc-8f52-4db9-96d6-de5e68d52eaa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:24:29.514 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:29.530+0000 7f08663ba640 10 librados: watch_flush enter 2024-03-29T11:24:29.514 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:29.530+0000 7f08663ba640 10 librados: watch_flush exit 2024-03-29T11:24:29.514 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:29.530+0000 7f08663ba640 20 client.4378.objecter shutdown clearing up homeless session... 2024-03-29T11:24:29.514 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:29.530+0000 7f08663ba640 10 client.4378.objecter successfully canceled tick 2024-03-29T11:24:29.514 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:29.531+0000 7f08663ba640 1 -- 172.21.15.84:0/3545294764 >> v1:172.21.15.179:6816/2981134980 conn(0x7f084006ddb0 legacy=0x7f0840070270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:29.514 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:29.531+0000 7f08663ba640 10 monclient: shutdown 2024-03-29T11:24:29.514 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:29.531+0000 7f08663ba640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:24:29.514 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:29.531+0000 7f08663ba640 1 -- 172.21.15.84:0/3545294764 >> v1:172.21.15.179:6789/0 conn(0x7f086013ee90 legacy=0x7f08601df440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:29.514 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:29.531+0000 7f08663ba640 1 -- 172.21.15.84:0/3545294764 shutdown_connections 2024-03-29T11:24:29.514 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:29.531+0000 7f08663ba640 1 -- 172.21.15.84:0/3545294764 >> 172.21.15.84:0/3545294764 conn(0x7f08600b35e0 msgr2=0x7f086013b190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:24:29.514 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:29.531+0000 7f08663ba640 1 -- 172.21.15.84:0/3545294764 shutdown_connections 2024-03-29T11:24:29.515 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:29.531+0000 7f08663ba640 1 -- 172.21.15.84:0/3545294764 wait complete. 2024-03-29T11:24:29.515 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:29.531+0000 7f08663ba640 1 librados: shutdown 2024-03-29T11:24:29.531 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:24:36.338 DEBUG:teuthology.orchestra.run.smithi084:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:24:36.485 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:36.503+0000 7fafacc63640 10 monclient: get_monmap_and_config 2024-03-29T11:24:36.486 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:36.503+0000 7fafacc63640 10 monclient: build_initial_monmap 2024-03-29T11:24:36.486 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:36.503+0000 7fafacc63640 10 monclient: monmap: 2024-03-29T11:24:36.486 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-29T11:24:36.486 INFO:teuthology.misc.health.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:24:36.486 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T11:24:36.504651+0000 2024-03-29T11:24:36.486 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T11:24:36.504651+0000 2024-03-29T11:24:36.486 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-29T11:24:36.486 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:24:36.486 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-29T11:24:36.486 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.noname-c 2024-03-29T11:24:36.486 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-b 2024-03-29T11:24:36.487 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:24:36.488 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:36.504+0000 7fafacc63640 1 Processor -- start 2024-03-29T11:24:36.488 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:36.504+0000 7fafacc63640 1 -- start start 2024-03-29T11:24:36.489 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:36.504+0000 7fafacc63640 10 monclient: init 2024-03-29T11:24:36.489 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:36.506+0000 7fafacc63640 10 monclient: _reopen_session rank -1 2024-03-29T11:24:36.489 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:36.506+0000 7fafacc63640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:24:36.489 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:36.506+0000 7fafacc63640 10 monclient(hunting): picked mon.noname-a con 0x7fafa80b1f60 addr v1:172.21.15.84:6789/0 2024-03-29T11:24:36.489 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:36.506+0000 7fafacc63640 10 monclient(hunting): picked mon.noname-c con 0x7fafa8071eb0 addr v1:172.21.15.179:6789/0 2024-03-29T11:24:36.489 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:36.506+0000 7fafacc63640 10 monclient(hunting): picked mon.noname-b con 0x7fafa8143d10 addr v1:172.21.15.84:6790/0 2024-03-29T11:24:36.489 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:36.506+0000 7fafacc63640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fafa806ef80 con 0x7fafa80b1f60 2024-03-29T11:24:36.489 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:36.506+0000 7fafacc63640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fafa806f260 con 0x7fafa8071eb0 2024-03-29T11:24:36.489 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:36.506+0000 7fafacc63640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fafa80b40c0 con 0x7fafa8143d10 2024-03-29T11:24:36.489 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:36.506+0000 7fafacc63640 10 monclient(hunting): _renew_subs 2024-03-29T11:24:36.489 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:36.506+0000 7fafacc63640 10 monclient(hunting): authenticate will time out at 1137.800703s 2024-03-29T11:24:36.493 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:36.511+0000 7fafa659c640 1 --1- >> v1:172.21.15.84:6789/0 conn(0x7fafa80b1f60 0x7fafa8143600 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.84:6789/0 says I am v1:172.21.15.84:52519/0 (socket says 172.21.15.84:52519) 2024-03-29T11:24:36.493 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:36.511+0000 7fafa659c640 1 -- 172.21.15.84:0/358895150 learned_addr learned my addr 172.21.15.84:0/358895150 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-29T11:24:36.494 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:36.511+0000 7fafa559a640 1 -- 172.21.15.84:0/358895150 <== mon.0 v1:172.21.15.84:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 644647844 0 0) 0x7fafa806ef80 con 0x7fafa80b1f60 2024-03-29T11:24:36.494 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:36.511+0000 7fafa559a640 10 monclient(hunting): _negotiate 2024-03-29T11:24:36.495 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:36.511+0000 7fafa559a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:36.495 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:36.511+0000 7fafa559a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:36.495 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:36.511+0000 7fafa559a640 10 monclient(hunting): my global_id is 4425 2024-03-29T11:24:36.495 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:36.511+0000 7fafa559a640 1 -- 172.21.15.84:0/358895150 --> v1:172.21.15.84:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faf90003520 con 0x7fafa80b1f60 2024-03-29T11:24:36.495 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:36.512+0000 7fafa559a640 1 -- 172.21.15.84:0/358895150 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1631005356 0 0) 0x7fafa80b40c0 con 0x7fafa8143d10 2024-03-29T11:24:36.495 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:36.512+0000 7fafa559a640 10 monclient(hunting): _negotiate 2024-03-29T11:24:36.495 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:36.512+0000 7fafa559a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:36.495 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:36.512+0000 7fafa559a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:36.495 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:36.512+0000 7fafa559a640 10 monclient(hunting): my global_id is 4421 2024-03-29T11:24:36.495 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:36.512+0000 7fafa559a640 1 -- 172.21.15.84:0/358895150 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fafa806ef80 con 0x7fafa8143d10 2024-03-29T11:24:36.499 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:36.512+0000 7fafa559a640 1 -- 172.21.15.84:0/358895150 <== mon.0 v1:172.21.15.84:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 214775644 0 0) 0x7faf90003520 con 0x7fafa80b1f60 2024-03-29T11:24:36.499 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:36.512+0000 7fafa559a640 1 -- 172.21.15.84:0/358895150 >> v1:172.21.15.84:6790/0 conn(0x7fafa8143d10 legacy=0x7fafa8146b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:36.499 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:36.512+0000 7fafa559a640 1 -- 172.21.15.84:0/358895150 >> v1:172.21.15.179:6789/0 conn(0x7fafa8071eb0 legacy=0x7fafa8146470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:36.499 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:36.512+0000 7fafa559a640 10 monclient: _finish_hunting 0 2024-03-29T11:24:36.499 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:36.512+0000 7fafa559a640 1 monclient: found mon.noname-a 2024-03-29T11:24:36.499 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:36.512+0000 7fafa559a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:24:36.499 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:36.512+0000 7fafa559a640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.84:6789/0 2024-03-29T11:24:36.499 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:36.512+0000 7fafa559a640 1 -- 172.21.15.84:0/358895150 --> v1:172.21.15.84:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fafa8155d70 con 0x7fafa80b1f60 2024-03-29T11:24:36.499 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:36.512+0000 7fafa559a640 10 monclient: _finish_auth 0 2024-03-29T11:24:36.499 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:36.512+0000 7fafa559a640 10 monclient: _check_auth_tickets 2024-03-29T11:24:36.499 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:36.512+0000 7fafa559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:24:36.499 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:36.512+0000 7fafacc63640 5 monclient: authenticate success, global_id 4425 2024-03-29T11:24:36.499 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:36.512+0000 7fafacc63640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:24:36.500 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:36.512+0000 7fafa559a640 1 -- 172.21.15.84:0/358895150 <== mon.0 v1:172.21.15.84:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4236868694 0 0) 0x7faf9c001690 con 0x7fafa80b1f60 2024-03-29T11:24:36.500 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:36.512+0000 7fafa559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:36.500 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:36.512+0000 7fafa559a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:24:36.500 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:36.512+0000 7fafa559a640 10 monclient: dump: 2024-03-29T11:24:36.500 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T11:24:36.500 INFO:teuthology.misc.health.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:24:36.500 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T11:18:53.864025+0000 2024-03-29T11:24:36.500 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T11:18:53.864025+0000 2024-03-29T11:24:36.500 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:36.500 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:24:36.500 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T11:24:36.500 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T11:24:36.500 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T11:24:36.500 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:24:36.500 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:36.512+0000 7fafacc63640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:24:36.500 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:36.514+0000 7fafa559a640 1 -- 172.21.15.84:0/358895150 <== mon.0 v1:172.21.15.84:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7faf9c0018b0 con 0x7fafa80b1f60 2024-03-29T11:24:36.501 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:36.514+0000 7fafa559a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:24:36.501 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:36.514+0000 7fafacc63640 10 monclient: get_monmap_and_config success 2024-03-29T11:24:36.501 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:36.514+0000 7fafacc63640 4 set_mon_vals no callback set 2024-03-29T11:24:36.501 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:36.514+0000 7fafacc63640 10 monclient: shutdown 2024-03-29T11:24:36.501 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:36.514+0000 7fafacc63640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:24:36.501 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:36.514+0000 7fafacc63640 1 -- 172.21.15.84:0/358895150 >> v1:172.21.15.84:6789/0 conn(0x7fafa80b1f60 legacy=0x7fafa8143600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:36.501 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:36.516+0000 7fafacc63640 1 -- 172.21.15.84:0/358895150 shutdown_connections 2024-03-29T11:24:36.501 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:36.516+0000 7fafacc63640 1 -- 172.21.15.84:0/358895150 >> 172.21.15.84:0/358895150 conn(0x7fafa8066a00 msgr2=0x7fafa8066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:24:36.501 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:36.516+0000 7fafacc63640 1 -- 172.21.15.84:0/358895150 shutdown_connections 2024-03-29T11:24:36.501 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:36.516+0000 7fafacc63640 1 -- 172.21.15.84:0/358895150 wait complete. 2024-03-29T11:24:36.503 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:36.517+0000 7fafacc63640 10 monclient: build_initial_monmap 2024-03-29T11:24:36.503 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:36.517+0000 7fafacc63640 10 monclient: monmap: 2024-03-29T11:24:36.503 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-29T11:24:36.504 INFO:teuthology.misc.health.smithi084.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:24:36.504 INFO:teuthology.misc.health.smithi084.stderr:last_changed 0.000000 2024-03-29T11:24:36.504 INFO:teuthology.misc.health.smithi084.stderr:created 0.000000 2024-03-29T11:24:36.504 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-29T11:24:36.504 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:24:36.504 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-29T11:24:36.504 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.noname-b 2024-03-29T11:24:36.504 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-c 2024-03-29T11:24:36.504 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:24:36.504 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:36.518+0000 7fafacc63640 1 librados: starting msgr at 2024-03-29T11:24:36.504 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:36.518+0000 7fafacc63640 1 librados: starting objecter 2024-03-29T11:24:36.504 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:36.518+0000 7fafacc63640 1 Processor -- start 2024-03-29T11:24:36.504 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:36.518+0000 7fafacc63640 1 -- start start 2024-03-29T11:24:36.504 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:36.518+0000 7fafacc63640 1 librados: setting wanted keys 2024-03-29T11:24:36.505 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:36.518+0000 7fafacc63640 1 librados: calling monclient init 2024-03-29T11:24:36.505 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:36.518+0000 7fafacc63640 10 monclient: init 2024-03-29T11:24:36.505 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:36.519+0000 7fafacc63640 10 monclient: _reopen_session rank -1 2024-03-29T11:24:36.505 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:36.519+0000 7fafacc63640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:24:36.505 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:36.519+0000 7fafacc63640 10 monclient(hunting): picked mon.noname-c con 0x7fafa80b1f60 addr v1:172.21.15.84:6790/0 2024-03-29T11:24:36.505 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:36.519+0000 7fafacc63640 10 monclient(hunting): picked mon.noname-b con 0x7fafa8143d10 addr v1:172.21.15.179:6789/0 2024-03-29T11:24:36.507 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:36.523+0000 7fafa6d9d640 1 --1- >> v1:172.21.15.84:6789/0 conn(0x7fafa8147ea0 0x7fafa813e1f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.84:6789/0 says I am v1:172.21.15.84:52715/0 (socket says 172.21.15.84:52715) 2024-03-29T11:24:36.507 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:36.523+0000 7fafa6d9d640 1 -- 172.21.15.84:0/1919703415 learned_addr learned my addr 172.21.15.84:0/1919703415 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-29T11:24:36.507 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:36.523+0000 7fafacc63640 10 monclient(hunting): picked mon.noname-a con 0x7fafa8147ea0 addr v1:172.21.15.84:6789/0 2024-03-29T11:24:36.507 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:36.523+0000 7fafacc63640 1 -- 172.21.15.84:0/1919703415 --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fafa8149770 con 0x7fafa8147ea0 2024-03-29T11:24:36.507 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:36.523+0000 7fafacc63640 1 -- 172.21.15.84:0/1919703415 --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fafa814b200 con 0x7fafa8143d10 2024-03-29T11:24:36.507 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:36.523+0000 7fafacc63640 1 -- 172.21.15.84:0/1919703415 --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fafa81499e0 con 0x7fafa80b1f60 2024-03-29T11:24:36.507 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:36.523+0000 7fafacc63640 10 monclient(hunting): _renew_subs 2024-03-29T11:24:36.507 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:36.523+0000 7fafacc63640 10 monclient(hunting): authenticate will time out at 1137.817504s 2024-03-29T11:24:36.507 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:36.524+0000 7faf96ffd640 10 client.?.objecter ms_handle_connect 0x7fafa8147ea0 2024-03-29T11:24:36.507 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:36.524+0000 7faf96ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:24:36.507 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:36.524+0000 7faf96ffd640 1 -- 172.21.15.84:0/1919703415 <== mon.0 v1:172.21.15.84:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3851058935 0 0) 0x7fafa8149770 con 0x7fafa8147ea0 2024-03-29T11:24:36.507 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:36.524+0000 7faf96ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:24:36.508 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:36.524+0000 7faf96ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:36.508 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:36.524+0000 7faf96ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:36.508 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:36.524+0000 7faf96ffd640 10 monclient(hunting): my global_id is 4428 2024-03-29T11:24:36.508 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:36.524+0000 7faf96ffd640 1 -- 172.21.15.84:0/1919703415 --> v1:172.21.15.84:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faf88004eb0 con 0x7fafa8147ea0 2024-03-29T11:24:36.509 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:36.527+0000 7faf96ffd640 1 -- 172.21.15.84:0/1919703415 <== mon.0 v1:172.21.15.84:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 51974860 0 0) 0x7faf88004eb0 con 0x7fafa8147ea0 2024-03-29T11:24:36.509 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:36.527+0000 7faf96ffd640 1 -- 172.21.15.84:0/1919703415 --> v1:172.21.15.84:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fafa8149770 con 0x7fafa8147ea0 2024-03-29T11:24:36.509 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:36.527+0000 7faf96ffd640 1 -- 172.21.15.84:0/1919703415 <== mon.0 v1:172.21.15.84:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4236868694 0 0) 0x7fafa0001c90 con 0x7fafa8147ea0 2024-03-29T11:24:36.509 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:36.527+0000 7faf96ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:36.509 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:36.527+0000 7faf96ffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:24:36.510 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:36.527+0000 7faf96ffd640 10 monclient(hunting): dump: 2024-03-29T11:24:36.510 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T11:24:36.510 INFO:teuthology.misc.health.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:24:36.510 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T11:18:53.864025+0000 2024-03-29T11:24:36.510 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T11:18:53.864025+0000 2024-03-29T11:24:36.510 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:36.510 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:24:36.510 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T11:24:36.510 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T11:24:36.510 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T11:24:36.510 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:24:36.510 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:36.527+0000 7faf96ffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:24:36.510 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:36.527+0000 7faf96ffd640 1 -- 172.21.15.84:0/1919703415 <== mon.0 v1:172.21.15.84:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2232179498 0 0) 0x7fafa8149770 con 0x7fafa8147ea0 2024-03-29T11:24:36.510 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:36.527+0000 7faf96ffd640 1 -- 172.21.15.84:0/1919703415 >> v1:172.21.15.84:6790/0 conn(0x7fafa80b1f60 legacy=0x7fafa8149060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:36.511 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:36.527+0000 7faf96ffd640 1 -- 172.21.15.84:0/1919703415 >> v1:172.21.15.179:6789/0 conn(0x7fafa8143d10 legacy=0x7fafa8147790 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:24:36.513 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:36.528+0000 7faf96ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:24:36.513 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:36.528+0000 7faf96ffd640 1 monclient: found mon.a 2024-03-29T11:24:36.513 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:36.528+0000 7faf96ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:24:36.513 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:36.528+0000 7faf96ffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.84:6789/0 2024-03-29T11:24:36.513 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:36.528+0000 7faf96ffd640 1 -- 172.21.15.84:0/1919703415 --> v1:172.21.15.84:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fafa8149f00 con 0x7fafa8147ea0 2024-03-29T11:24:36.514 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:36.528+0000 7faf96ffd640 10 monclient: _finish_auth 0 2024-03-29T11:24:36.514 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:36.528+0000 7faf96ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:24:36.514 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:36.528+0000 7faf96ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:24:36.514 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:36.528+0000 7fafacc63640 5 monclient: authenticate success, global_id 4428 2024-03-29T11:24:36.514 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:36.528+0000 7fafacc63640 10 monclient: _renew_subs 2024-03-29T11:24:36.514 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:36.528+0000 7fafacc63640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.84:6789/0 2024-03-29T11:24:36.514 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:36.528+0000 7fafacc63640 1 -- 172.21.15.84:0/1919703415 --> v1:172.21.15.84:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fafa8154720 con 0x7fafa8147ea0 2024-03-29T11:24:36.514 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:36.528+0000 7fafacc63640 10 client.4428.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:24:36.514 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:36.528+0000 7fafacc63640 10 monclient: _renew_subs 2024-03-29T11:24:36.514 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:36.528+0000 7fafacc63640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.84:6789/0 2024-03-29T11:24:36.514 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:36.528+0000 7fafacc63640 1 -- 172.21.15.84:0/1919703415 --> v1:172.21.15.84:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fafa813e900 con 0x7fafa8147ea0 2024-03-29T11:24:36.514 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:36.528+0000 7fafacc63640 1 librados: init done 2024-03-29T11:24:36.514 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:36.529+0000 7faf96ffd640 1 -- 172.21.15.84:0/1919703415 <== mon.0 v1:172.21.15.84:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fafa00020c0 con 0x7fafa8147ea0 2024-03-29T11:24:36.515 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:36.529+0000 7faf96ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:24:36.515 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:36.529+0000 7faf96ffd640 1 -- 172.21.15.84:0/1919703415 <== mon.0 v1:172.21.15.84:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4236868694 0 0) 0x7fafa0003de0 con 0x7fafa8147ea0 2024-03-29T11:24:36.515 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:36.529+0000 7faf96ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:36.515 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:36.529+0000 7faf96ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:24:36.515 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:36.529+0000 7faf96ffd640 10 monclient: dump: 2024-03-29T11:24:36.515 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T11:24:36.515 INFO:teuthology.misc.health.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:24:36.515 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T11:18:53.864025+0000 2024-03-29T11:24:36.515 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T11:18:53.864025+0000 2024-03-29T11:24:36.515 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:36.515 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:24:36.515 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T11:24:36.515 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T11:24:36.515 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T11:24:36.515 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:24:36.516 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:36.530+0000 7fafacc63640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:24:36.516 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:36.530+0000 7fafacc63640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:24:36.516 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:36.530+0000 7fafacc63640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.84:6789/0 2024-03-29T11:24:36.516 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:36.530+0000 7fafacc63640 1 -- 172.21.15.84:0/1919703415 --> v1:172.21.15.84:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fafa8149b70 con 0x7fafa8147ea0 2024-03-29T11:24:36.516 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:36.531+0000 7fafa4d99640 4 set_mon_vals no callback set 2024-03-29T11:24:36.517 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:36.532+0000 7faf96ffd640 1 -- 172.21.15.84:0/1919703415 <== mon.0 v1:172.21.15.84:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4033942909 0 0) 0x7fafa0003f40 con 0x7fafa8147ea0 2024-03-29T11:24:36.517 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:36.532+0000 7faf96ffd640 1 -- 172.21.15.84:0/1919703415 <== mon.0 v1:172.21.15.84:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2808139587 0 0) 0x7fafa0084d50 con 0x7fafa8147ea0 2024-03-29T11:24:36.517 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:36.532+0000 7faf96ffd640 10 client.4428.objecter ms_dispatch 0x7fafa8000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:24:36.517 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:36.532+0000 7faf96ffd640 3 client.4428.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:24:36.517 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:36.532+0000 7faf96ffd640 3 client.4428.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:24:36.517 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:36.532+0000 7faf96ffd640 20 client.4428.objecter dump_active .. 0 homeless 2024-03-29T11:24:36.517 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:36.533+0000 7fafa659c640 10 monclient: get_auth_request con 0x7faf8806dcd0 auth_method 0 2024-03-29T11:24:36.517 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:36.533+0000 7faf96ffd640 10 client.4428.objecter ms_handle_connect 0x7faf8806dcd0 2024-03-29T11:24:36.521 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:36.539+0000 7faf96ffd640 1 -- 172.21.15.84:0/1919703415 <== mon.0 v1:172.21.15.84:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fafa0050630 con 0x7fafa8147ea0 2024-03-29T11:24:36.521 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:36.539+0000 7faf96ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:24:36.523 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:36.539+0000 7faf96ffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:24:36.842 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:36.860+0000 7fafacc63640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:24:36.843 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:36.860+0000 7fafacc63640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:24:36.843 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:36.860+0000 7fafacc63640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.84:6789/0 2024-03-29T11:24:36.843 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:36.860+0000 7fafacc63640 1 -- 172.21.15.84:0/1919703415 --> v1:172.21.15.84:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fafa814b000 con 0x7fafa8147ea0 2024-03-29T11:24:36.844 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:36.862+0000 7faf96ffd640 1 -- 172.21.15.84:0/1919703415 <== mon.0 v1:172.21.15.84:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4209477549) 0x7fafa0052560 con 0x7fafa8147ea0 2024-03-29T11:24:36.844 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:36.862+0000 7faf96ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:24:36.844 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:36.862+0000 7faf96ffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:24:36.844 INFO:teuthology.misc.health.smithi084.stdout: 2024-03-29T11:24:36.844 INFO:teuthology.misc.health.smithi084.stdout:{"epoch":10,"fsid":"2c3e002f-9eb0-4860-853b-4ef1ef562213","created":"2024-03-29T11:19:32.029169+0000","modified":"2024-03-29T11:19:34.200299+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:34.200299+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"5c39e276-2034-4716-bbc0-57a634eefa92","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"cd580e99-109f-4bf1-88c0-a5987bc8b9ed","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"11e60a5a-051d-453f-a190-66387cb35592","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"2c329fbe-bbc4-4030-bf87-d711cad04679","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f4213d4a-bd36-4433-91ae-d7e2b02481d3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"f0cf44ea-6055-48af-805b-aed858102ff9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"d895eaa0-cc0b-4ee4-9b5a-2dfcd89a60b7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"9bc5a7bc-8f52-4db9-96d6-de5e68d52eaa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:24:36.847 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:36.865+0000 7fafacc63640 10 librados: watch_flush enter 2024-03-29T11:24:36.850 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:36.866+0000 7fafacc63640 10 librados: watch_flush exit 2024-03-29T11:24:36.850 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:36.866+0000 7fafacc63640 20 client.4428.objecter shutdown clearing up homeless session... 2024-03-29T11:24:36.850 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:36.866+0000 7fafacc63640 10 client.4428.objecter successfully canceled tick 2024-03-29T11:24:36.850 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:36.866+0000 7fafacc63640 1 -- 172.21.15.84:0/1919703415 >> v1:172.21.15.179:6816/2981134980 conn(0x7faf8806dcd0 legacy=0x7faf88070190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:36.850 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:36.866+0000 7fafacc63640 10 monclient: shutdown 2024-03-29T11:24:36.850 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:36.866+0000 7fafacc63640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:24:36.850 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:36.866+0000 7fafacc63640 1 -- 172.21.15.84:0/1919703415 >> v1:172.21.15.84:6789/0 conn(0x7fafa8147ea0 legacy=0x7fafa813e1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:36.850 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:36.866+0000 7fafacc63640 1 -- 172.21.15.84:0/1919703415 shutdown_connections 2024-03-29T11:24:36.850 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:36.866+0000 7fafacc63640 1 -- 172.21.15.84:0/1919703415 >> 172.21.15.84:0/1919703415 conn(0x7fafa8066a00 msgr2=0x7fafa8142a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:24:36.850 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:36.867+0000 7fafacc63640 1 -- 172.21.15.84:0/1919703415 shutdown_connections 2024-03-29T11:24:36.851 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:36.868+0000 7fafacc63640 1 -- 172.21.15.84:0/1919703415 wait complete. 2024-03-29T11:24:36.851 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:36.868+0000 7fafacc63640 1 librados: shutdown 2024-03-29T11:24:36.868 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:24:43.675 DEBUG:teuthology.orchestra.run.smithi084:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:24:43.831 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:43.849+0000 7f5d3350e640 10 monclient: get_monmap_and_config 2024-03-29T11:24:43.832 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:43.849+0000 7f5d3350e640 10 monclient: build_initial_monmap 2024-03-29T11:24:43.832 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:43.849+0000 7f5d3350e640 10 monclient: monmap: 2024-03-29T11:24:43.832 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-29T11:24:43.832 INFO:teuthology.misc.health.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:24:43.832 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T11:24:43.850549+0000 2024-03-29T11:24:43.832 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T11:24:43.850549+0000 2024-03-29T11:24:43.832 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-29T11:24:43.832 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:24:43.832 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-29T11:24:43.832 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.noname-c 2024-03-29T11:24:43.832 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-b 2024-03-29T11:24:43.832 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:24:43.833 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:43.850+0000 7f5d3350e640 1 Processor -- start 2024-03-29T11:24:43.833 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:43.850+0000 7f5d3350e640 1 -- start start 2024-03-29T11:24:43.833 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:43.850+0000 7f5d3350e640 10 monclient: init 2024-03-29T11:24:43.835 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:43.851+0000 7f5d3350e640 10 monclient: _reopen_session rank -1 2024-03-29T11:24:43.835 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:43.851+0000 7f5d3350e640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:24:43.835 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:43.851+0000 7f5d3350e640 10 monclient(hunting): picked mon.noname-a con 0x7f5d2c1457f0 addr v1:172.21.15.84:6789/0 2024-03-29T11:24:43.835 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:43.851+0000 7f5d3350e640 10 monclient(hunting): picked mon.noname-b con 0x7f5d2c13f650 addr v1:172.21.15.84:6790/0 2024-03-29T11:24:43.835 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:43.851+0000 7f5d3350e640 10 monclient(hunting): picked mon.noname-c con 0x7f5d2c0a9d70 addr v1:172.21.15.179:6789/0 2024-03-29T11:24:43.836 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:43.851+0000 7f5d3350e640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5d2c0aaef0 con 0x7f5d2c1457f0 2024-03-29T11:24:43.836 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:43.851+0000 7f5d3350e640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5d2c145bd0 con 0x7f5d2c0a9d70 2024-03-29T11:24:43.836 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:43.851+0000 7f5d3350e640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5d2c145eb0 con 0x7f5d2c13f650 2024-03-29T11:24:43.836 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:43.851+0000 7f5d3350e640 10 monclient(hunting): _renew_subs 2024-03-29T11:24:43.836 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:43.851+0000 7f5d3350e640 10 monclient(hunting): authenticate will time out at 1145.145815s 2024-03-29T11:24:43.837 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:43.854+0000 7f5d30aa9640 1 --1- >> v1:172.21.15.84:6790/0 conn(0x7f5d2c13f650 0x7f5d2c0b2620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.84:6790/0 says I am v1:172.21.15.84:37373/0 (socket says 172.21.15.84:37373) 2024-03-29T11:24:43.837 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:43.854+0000 7f5d30aa9640 1 -- 172.21.15.84:0/2775379851 learned_addr learned my addr 172.21.15.84:0/2775379851 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-29T11:24:43.837 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:43.854+0000 7f5d23fff640 1 -- 172.21.15.84:0/2775379851 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1561046854 0 0) 0x7f5d2c145eb0 con 0x7f5d2c13f650 2024-03-29T11:24:43.837 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:43.854+0000 7f5d23fff640 10 monclient(hunting): _negotiate 2024-03-29T11:24:43.837 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:43.854+0000 7f5d23fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:43.837 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:43.854+0000 7f5d23fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:43.838 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:43.854+0000 7f5d23fff640 10 monclient(hunting): my global_id is 4424 2024-03-29T11:24:43.838 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:43.854+0000 7f5d23fff640 1 -- 172.21.15.84:0/2775379851 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5d18003520 con 0x7f5d2c13f650 2024-03-29T11:24:43.838 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:43.855+0000 7f5d23fff640 1 -- 172.21.15.84:0/2775379851 <== mon.2 v1:172.21.15.84:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2561197861 0 0) 0x7f5d18003520 con 0x7f5d2c13f650 2024-03-29T11:24:43.838 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:43.855+0000 7f5d23fff640 1 -- 172.21.15.84:0/2775379851 >> v1:172.21.15.179:6789/0 conn(0x7f5d2c0a9d70 legacy=0x7f5d2c14d8c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:24:43.839 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:43.857+0000 7f5d23fff640 1 -- 172.21.15.84:0/2775379851 >> v1:172.21.15.84:6789/0 conn(0x7f5d2c1457f0 legacy=0x7f5d2c13ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:24:43.841 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:43.857+0000 7f5d23fff640 10 monclient: _finish_hunting 0 2024-03-29T11:24:43.841 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:43.857+0000 7f5d23fff640 1 monclient: found mon.noname-b 2024-03-29T11:24:43.841 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:43.857+0000 7f5d23fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:24:43.841 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:43.857+0000 7f5d23fff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.84:6790/0 2024-03-29T11:24:43.841 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:43.857+0000 7f5d23fff640 1 -- 172.21.15.84:0/2775379851 --> v1:172.21.15.84:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d2c146340 con 0x7f5d2c13f650 2024-03-29T11:24:43.841 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:43.857+0000 7f5d23fff640 10 monclient: _finish_auth 0 2024-03-29T11:24:43.841 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:43.857+0000 7f5d23fff640 10 monclient: _check_auth_tickets 2024-03-29T11:24:43.841 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:43.857+0000 7f5d23fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:24:43.841 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:43.857+0000 7f5d23fff640 1 -- 172.21.15.84:0/2775379851 <== mon.2 v1:172.21.15.84:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4236868694 0 0) 0x7f5d1c001690 con 0x7f5d2c13f650 2024-03-29T11:24:43.841 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:43.857+0000 7f5d23fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:43.841 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:43.857+0000 7f5d23fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:24:43.841 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:43.857+0000 7f5d23fff640 10 monclient: dump: 2024-03-29T11:24:43.841 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T11:24:43.842 INFO:teuthology.misc.health.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:24:43.842 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T11:18:53.864025+0000 2024-03-29T11:24:43.842 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T11:18:53.864025+0000 2024-03-29T11:24:43.842 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:43.842 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:24:43.842 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T11:24:43.842 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T11:24:43.842 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T11:24:43.842 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:24:43.842 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:43.858+0000 7f5d3350e640 5 monclient: authenticate success, global_id 4424 2024-03-29T11:24:43.842 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:43.858+0000 7f5d3350e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:24:43.842 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:43.858+0000 7f5d23fff640 1 -- 172.21.15.84:0/2775379851 <== mon.2 v1:172.21.15.84:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5d1c001860 con 0x7f5d2c13f650 2024-03-29T11:24:43.842 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:43.858+0000 7f5d23fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:24:43.842 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:43.858+0000 7f5d23fff640 1 -- 172.21.15.84:0/2775379851 <== mon.2 v1:172.21.15.84:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4236868694 0 0) 0x7f5d1c001c70 con 0x7f5d2c13f650 2024-03-29T11:24:43.843 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:43.858+0000 7f5d23fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:43.843 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:43.858+0000 7f5d23fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:24:43.843 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:43.858+0000 7f5d23fff640 10 monclient: dump: 2024-03-29T11:24:43.843 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T11:24:43.843 INFO:teuthology.misc.health.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:24:43.843 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T11:18:53.864025+0000 2024-03-29T11:24:43.843 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T11:18:53.864025+0000 2024-03-29T11:24:43.843 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:43.843 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:24:43.843 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T11:24:43.843 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T11:24:43.843 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T11:24:43.843 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:24:43.845 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:43.859+0000 7f5d3350e640 10 monclient: get_monmap_and_config success 2024-03-29T11:24:43.845 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:43.859+0000 7f5d3350e640 4 set_mon_vals no callback set 2024-03-29T11:24:43.845 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:43.860+0000 7f5d3350e640 10 monclient: shutdown 2024-03-29T11:24:43.845 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:43.860+0000 7f5d3350e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:24:43.845 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:43.860+0000 7f5d3350e640 1 -- 172.21.15.84:0/2775379851 >> v1:172.21.15.84:6790/0 conn(0x7f5d2c13f650 legacy=0x7f5d2c0b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:43.845 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:43.860+0000 7f5d3350e640 1 -- 172.21.15.84:0/2775379851 shutdown_connections 2024-03-29T11:24:43.845 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:43.860+0000 7f5d3350e640 1 -- 172.21.15.84:0/2775379851 >> 172.21.15.84:0/2775379851 conn(0x7f5d2c066a00 msgr2=0x7f5d2c066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:24:43.845 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:43.861+0000 7f5d3350e640 1 -- 172.21.15.84:0/2775379851 shutdown_connections 2024-03-29T11:24:43.845 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:43.861+0000 7f5d3350e640 1 -- 172.21.15.84:0/2775379851 wait complete. 2024-03-29T11:24:43.845 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:43.862+0000 7f5d3350e640 10 monclient: build_initial_monmap 2024-03-29T11:24:43.845 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:43.862+0000 7f5d3350e640 10 monclient: monmap: 2024-03-29T11:24:43.846 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-29T11:24:43.846 INFO:teuthology.misc.health.smithi084.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:24:43.846 INFO:teuthology.misc.health.smithi084.stderr:last_changed 0.000000 2024-03-29T11:24:43.846 INFO:teuthology.misc.health.smithi084.stderr:created 0.000000 2024-03-29T11:24:43.846 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-29T11:24:43.846 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:24:43.846 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-29T11:24:43.846 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.noname-b 2024-03-29T11:24:43.846 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-c 2024-03-29T11:24:43.846 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:24:43.846 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:43.862+0000 7f5d3350e640 1 librados: starting msgr at 2024-03-29T11:24:43.846 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:43.862+0000 7f5d3350e640 1 librados: starting objecter 2024-03-29T11:24:43.846 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:43.862+0000 7f5d3350e640 1 Processor -- start 2024-03-29T11:24:43.847 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:43.863+0000 7f5d3350e640 1 -- start start 2024-03-29T11:24:43.847 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:43.863+0000 7f5d3350e640 1 librados: setting wanted keys 2024-03-29T11:24:43.847 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:43.863+0000 7f5d3350e640 1 librados: calling monclient init 2024-03-29T11:24:43.848 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:43.863+0000 7f5d3350e640 10 monclient: init 2024-03-29T11:24:43.848 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:43.863+0000 7f5d3350e640 10 monclient: _reopen_session rank -1 2024-03-29T11:24:43.848 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:43.863+0000 7f5d3350e640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:24:43.848 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:43.863+0000 7f5d3350e640 10 monclient(hunting): picked mon.noname-b con 0x7f5d2c0a9d70 addr v1:172.21.15.179:6789/0 2024-03-29T11:24:43.848 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:43.863+0000 7f5d3350e640 10 monclient(hunting): picked mon.noname-c con 0x7f5d2c13f650 addr v1:172.21.15.84:6790/0 2024-03-29T11:24:43.849 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:43.864+0000 7f5d3350e640 10 monclient(hunting): picked mon.noname-a con 0x7f5d2c1457f0 addr v1:172.21.15.84:6789/0 2024-03-29T11:24:43.850 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:43.864+0000 7f5d3350e640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5d2c13a670 con 0x7f5d2c1457f0 2024-03-29T11:24:43.850 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:43.864+0000 7f5d3350e640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5d2c142780 con 0x7f5d2c0a9d70 2024-03-29T11:24:43.850 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:43.864+0000 7f5d3350e640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5d2c139470 con 0x7f5d2c13f650 2024-03-29T11:24:43.850 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:43.864+0000 7f5d3350e640 10 monclient(hunting): _renew_subs 2024-03-29T11:24:43.850 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:43.864+0000 7f5d3350e640 10 monclient(hunting): authenticate will time out at 1145.159004s 2024-03-29T11:24:43.850 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:43.867+0000 7f5d30aa9640 1 --1- >> v1:172.21.15.84:6790/0 conn(0x7f5d2c13f650 0x7f5d2c13ad70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.84:6790/0 says I am v1:172.21.15.84:37431/0 (socket says 172.21.15.84:37431) 2024-03-29T11:24:43.850 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:43.867+0000 7f5d30aa9640 1 -- 172.21.15.84:0/4283857028 learned_addr learned my addr 172.21.15.84:0/4283857028 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-29T11:24:43.851 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:43.869+0000 7f5d21ffb640 10 client.?.objecter ms_handle_connect 0x7f5d2c13f650 2024-03-29T11:24:43.851 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:43.869+0000 7f5d21ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:24:43.852 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:43.870+0000 7f5d21ffb640 10 client.?.objecter ms_handle_connect 0x7f5d2c0a9d70 2024-03-29T11:24:43.852 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:43.870+0000 7f5d21ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:24:43.852 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:43.870+0000 7f5d21ffb640 1 -- 172.21.15.84:0/4283857028 <== mon.1 v1:172.21.15.179:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1900928965 0 0) 0x7f5d2c142780 con 0x7f5d2c0a9d70 2024-03-29T11:24:43.852 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:43.870+0000 7f5d21ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:24:43.852 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:43.870+0000 7f5d21ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:43.853 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:43.870+0000 7f5d21ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:43.853 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:43.870+0000 7f5d21ffb640 10 monclient(hunting): my global_id is 4384 2024-03-29T11:24:43.853 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:43.870+0000 7f5d21ffb640 1 -- 172.21.15.84:0/4283857028 --> v1:172.21.15.179:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5d10004eb0 con 0x7f5d2c0a9d70 2024-03-29T11:24:43.853 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:43.871+0000 7f5d21ffb640 1 -- 172.21.15.84:0/4283857028 <== mon.1 v1:172.21.15.179:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 417671772 0 0) 0x7f5d10004eb0 con 0x7f5d2c0a9d70 2024-03-29T11:24:43.853 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:43.871+0000 7f5d21ffb640 1 -- 172.21.15.84:0/4283857028 --> v1:172.21.15.179:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5d10005300 con 0x7f5d2c0a9d70 2024-03-29T11:24:43.853 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:43.871+0000 7f5d21ffb640 1 -- 172.21.15.84:0/4283857028 <== mon.1 v1:172.21.15.179:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4236868694 0 0) 0x7f5d240017c0 con 0x7f5d2c0a9d70 2024-03-29T11:24:43.853 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:43.871+0000 7f5d21ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:43.853 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:43.871+0000 7f5d21ffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:24:43.853 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:43.871+0000 7f5d21ffb640 10 monclient(hunting): dump: 2024-03-29T11:24:43.854 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T11:24:43.854 INFO:teuthology.misc.health.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:24:43.854 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T11:18:53.864025+0000 2024-03-29T11:24:43.854 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T11:18:53.864025+0000 2024-03-29T11:24:43.854 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:43.854 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:24:43.854 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T11:24:43.854 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T11:24:43.854 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T11:24:43.854 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:24:43.854 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:43.871+0000 7f5d21ffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:24:43.855 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:43.871+0000 7f5d21ffb640 1 -- 172.21.15.84:0/4283857028 <== mon.1 v1:172.21.15.179:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 313217860 0 0) 0x7f5d10005300 con 0x7f5d2c0a9d70 2024-03-29T11:24:43.855 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:43.871+0000 7f5d21ffb640 1 -- 172.21.15.84:0/4283857028 >> v1:172.21.15.84:6790/0 conn(0x7f5d2c13f650 legacy=0x7f5d2c13ad70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:43.856 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:43.871+0000 7f5d21ffb640 1 -- 172.21.15.84:0/4283857028 >> v1:172.21.15.84:6789/0 conn(0x7f5d2c1457f0 legacy=0x7f5d2c1e5e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:43.856 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:43.871+0000 7f5d21ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:24:43.856 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:43.871+0000 7f5d21ffb640 1 monclient: found mon.b 2024-03-29T11:24:43.856 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:43.871+0000 7f5d21ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:24:43.856 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:43.871+0000 7f5d21ffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-29T11:24:43.856 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:43.871+0000 7f5d21ffb640 1 -- 172.21.15.84:0/4283857028 --> v1:172.21.15.179:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d2c142c70 con 0x7f5d2c0a9d70 2024-03-29T11:24:43.856 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:43.871+0000 7f5d21ffb640 10 monclient: _finish_auth 0 2024-03-29T11:24:43.856 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:43.871+0000 7f5d21ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:24:43.856 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:43.871+0000 7f5d21ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:24:43.856 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:43.872+0000 7f5d3350e640 5 monclient: authenticate success, global_id 4384 2024-03-29T11:24:43.856 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:43.872+0000 7f5d3350e640 10 monclient: _renew_subs 2024-03-29T11:24:43.856 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:43.872+0000 7f5d3350e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-29T11:24:43.856 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:43.872+0000 7f5d3350e640 1 -- 172.21.15.84:0/4283857028 --> v1:172.21.15.179:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5d2c13b480 con 0x7f5d2c0a9d70 2024-03-29T11:24:43.856 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:43.872+0000 7f5d3350e640 10 client.4384.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:24:43.857 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:43.872+0000 7f5d3350e640 10 monclient: _renew_subs 2024-03-29T11:24:43.857 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:43.872+0000 7f5d3350e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-29T11:24:43.857 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:43.872+0000 7f5d3350e640 1 -- 172.21.15.84:0/4283857028 --> v1:172.21.15.179:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f5d2c13b7d0 con 0x7f5d2c0a9d70 2024-03-29T11:24:43.857 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:43.872+0000 7f5d3350e640 1 librados: init done 2024-03-29T11:24:43.857 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:43.875+0000 7f5d177fe640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:24:43.857 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:43.875+0000 7f5d177fe640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:24:43.857 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:43.875+0000 7f5d177fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-29T11:24:43.857 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:43.875+0000 7f5d177fe640 1 -- 172.21.15.84:0/4283857028 --> v1:172.21.15.179:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5d2c135b50 con 0x7f5d2c0a9d70 2024-03-29T11:24:43.859 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:43.877+0000 7f5d21ffb640 1 -- 172.21.15.84:0/4283857028 <== mon.1 v1:172.21.15.179:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5d24001d30 con 0x7f5d2c0a9d70 2024-03-29T11:24:43.859 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:43.877+0000 7f5d21ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:24:43.859 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:43.877+0000 7f5d21ffb640 1 -- 172.21.15.84:0/4283857028 <== mon.1 v1:172.21.15.179:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4236868694 0 0) 0x7f5d240017c0 con 0x7f5d2c0a9d70 2024-03-29T11:24:43.859 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:43.877+0000 7f5d21ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:43.859 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:43.877+0000 7f5d21ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:24:43.859 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:43.877+0000 7f5d21ffb640 10 monclient: dump: 2024-03-29T11:24:43.859 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T11:24:43.859 INFO:teuthology.misc.health.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:24:43.859 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T11:18:53.864025+0000 2024-03-29T11:24:43.859 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T11:18:53.864025+0000 2024-03-29T11:24:43.860 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:43.860 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:24:43.860 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T11:24:43.860 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T11:24:43.860 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T11:24:43.860 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:24:43.860 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:43.877+0000 7f5d237fe640 4 set_mon_vals no callback set 2024-03-29T11:24:43.860 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:43.878+0000 7f5d21ffb640 1 -- 172.21.15.84:0/4283857028 <== mon.1 v1:172.21.15.179:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4033942909 0 0) 0x7f5d240195a0 con 0x7f5d2c0a9d70 2024-03-29T11:24:43.860 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:43.878+0000 7f5d30aa9640 10 monclient: get_auth_request con 0x7f5d1006e130 auth_method 0 2024-03-29T11:24:43.861 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:43.879+0000 7f5d21ffb640 1 -- 172.21.15.84:0/4283857028 <== mon.1 v1:172.21.15.179:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2808139587 0 0) 0x7f5d24053200 con 0x7f5d2c0a9d70 2024-03-29T11:24:43.861 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:43.879+0000 7f5d21ffb640 10 client.4384.objecter ms_dispatch 0x7f5d2c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:24:43.861 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:43.879+0000 7f5d21ffb640 3 client.4384.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:24:43.862 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:43.879+0000 7f5d21ffb640 3 client.4384.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:24:43.862 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:43.879+0000 7f5d21ffb640 20 client.4384.objecter dump_active .. 0 homeless 2024-03-29T11:24:43.862 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:43.879+0000 7f5d21ffb640 10 client.4384.objecter ms_handle_connect 0x7f5d1006e130 2024-03-29T11:24:43.867 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:43.885+0000 7f5d21ffb640 1 -- 172.21.15.84:0/4283857028 <== mon.1 v1:172.21.15.179:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f5d2404f4a0 con 0x7f5d2c0a9d70 2024-03-29T11:24:43.867 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:43.885+0000 7f5d21ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:24:43.867 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:43.885+0000 7f5d21ffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:24:44.190 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:44.205+0000 7f5d177fe640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:24:44.190 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:44.205+0000 7f5d177fe640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:24:44.190 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:44.205+0000 7f5d177fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-29T11:24:44.191 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:44.205+0000 7f5d177fe640 1 -- 172.21.15.84:0/4283857028 --> v1:172.21.15.179:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5d2c075f10 con 0x7f5d2c0a9d70 2024-03-29T11:24:44.192 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:44.210+0000 7f5d21ffb640 1 -- 172.21.15.84:0/4283857028 <== mon.1 v1:172.21.15.179:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4209477549) 0x7f5d2404f4a0 con 0x7f5d2c0a9d70 2024-03-29T11:24:44.192 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:44.210+0000 7f5d21ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:24:44.192 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:44.210+0000 7f5d21ffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:24:44.192 INFO:teuthology.misc.health.smithi084.stdout: 2024-03-29T11:24:44.192 INFO:teuthology.misc.health.smithi084.stdout:{"epoch":10,"fsid":"2c3e002f-9eb0-4860-853b-4ef1ef562213","created":"2024-03-29T11:19:32.029169+0000","modified":"2024-03-29T11:19:34.200299+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:34.200299+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"5c39e276-2034-4716-bbc0-57a634eefa92","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"cd580e99-109f-4bf1-88c0-a5987bc8b9ed","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"11e60a5a-051d-453f-a190-66387cb35592","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"2c329fbe-bbc4-4030-bf87-d711cad04679","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f4213d4a-bd36-4433-91ae-d7e2b02481d3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"f0cf44ea-6055-48af-805b-aed858102ff9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"d895eaa0-cc0b-4ee4-9b5a-2dfcd89a60b7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"9bc5a7bc-8f52-4db9-96d6-de5e68d52eaa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:24:44.195 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:44.213+0000 7f5d177fe640 10 librados: watch_flush enter 2024-03-29T11:24:44.195 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:44.213+0000 7f5d177fe640 10 librados: watch_flush exit 2024-03-29T11:24:44.195 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:44.213+0000 7f5d177fe640 20 client.4384.objecter shutdown clearing up homeless session... 2024-03-29T11:24:44.195 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:44.213+0000 7f5d177fe640 10 client.4384.objecter successfully canceled tick 2024-03-29T11:24:44.197 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:44.214+0000 7f5d177fe640 1 -- 172.21.15.84:0/4283857028 >> v1:172.21.15.179:6816/2981134980 conn(0x7f5d1006e130 legacy=0x7f5d100705f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:44.197 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:44.214+0000 7f5d177fe640 10 monclient: shutdown 2024-03-29T11:24:44.197 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:44.214+0000 7f5d177fe640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:24:44.197 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:44.215+0000 7f5d177fe640 1 -- 172.21.15.84:0/4283857028 >> v1:172.21.15.179:6789/0 conn(0x7f5d2c0a9d70 legacy=0x7f5d2c138d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:44.199 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:44.217+0000 7f5d177fe640 1 -- 172.21.15.84:0/4283857028 shutdown_connections 2024-03-29T11:24:44.199 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:44.217+0000 7f5d177fe640 1 -- 172.21.15.84:0/4283857028 >> 172.21.15.84:0/4283857028 conn(0x7f5d2c066a00 msgr2=0x7f5d2c0b3a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:24:44.200 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:44.217+0000 7f5d177fe640 1 -- 172.21.15.84:0/4283857028 shutdown_connections 2024-03-29T11:24:44.200 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:44.217+0000 7f5d177fe640 1 -- 172.21.15.84:0/4283857028 wait complete. 2024-03-29T11:24:44.200 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:44.218+0000 7f5d177fe640 1 librados: shutdown 2024-03-29T11:24:44.216 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:24:51.023 DEBUG:teuthology.orchestra.run.smithi084:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:24:51.196 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:51.213+0000 7f06a66d6640 10 monclient: get_monmap_and_config 2024-03-29T11:24:51.196 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:51.213+0000 7f06a66d6640 10 monclient: build_initial_monmap 2024-03-29T11:24:51.196 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:51.213+0000 7f06a66d6640 10 monclient: monmap: 2024-03-29T11:24:51.196 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-29T11:24:51.196 INFO:teuthology.misc.health.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:24:51.196 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T11:24:51.214549+0000 2024-03-29T11:24:51.196 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T11:24:51.214549+0000 2024-03-29T11:24:51.196 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-29T11:24:51.196 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:24:51.196 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-29T11:24:51.196 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.noname-c 2024-03-29T11:24:51.197 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-b 2024-03-29T11:24:51.197 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:24:51.200 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:51.214+0000 7f06a66d6640 1 Processor -- start 2024-03-29T11:24:51.200 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:51.215+0000 7f06a66d6640 1 -- start start 2024-03-29T11:24:51.200 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:51.215+0000 7f06a66d6640 10 monclient: init 2024-03-29T11:24:51.200 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:51.215+0000 7f06a66d6640 10 monclient: _reopen_session rank -1 2024-03-29T11:24:51.200 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:51.215+0000 7f06a66d6640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:24:51.200 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:51.215+0000 7f06a66d6640 10 monclient(hunting): picked mon.noname-b con 0x7f06a00b4330 addr v1:172.21.15.84:6790/0 2024-03-29T11:24:51.200 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:51.215+0000 7f06a66d6640 10 monclient(hunting): picked mon.noname-a con 0x7f06a01470b0 addr v1:172.21.15.84:6789/0 2024-03-29T11:24:51.200 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:51.215+0000 7f06a66d6640 10 monclient(hunting): picked mon.noname-c con 0x7f06a01439b0 addr v1:172.21.15.179:6789/0 2024-03-29T11:24:51.200 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:51.215+0000 7f06a66d6640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f06a006f2a0 con 0x7f06a01470b0 2024-03-29T11:24:51.201 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:51.215+0000 7f06a66d6640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f06a0146b90 con 0x7f06a01439b0 2024-03-29T11:24:51.201 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:51.215+0000 7f06a66d6640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f06a00b2ca0 con 0x7f06a00b4330 2024-03-29T11:24:51.201 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:51.215+0000 7f06a66d6640 10 monclient(hunting): _renew_subs 2024-03-29T11:24:51.201 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:51.215+0000 7f06a66d6640 10 monclient(hunting): authenticate will time out at 1152.509991s 2024-03-29T11:24:51.204 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:51.221+0000 7f069f7fe640 1 --1- >> v1:172.21.15.84:6789/0 conn(0x7f06a01470b0 0x7f06a00b2590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.84:6789/0 says I am v1:172.21.15.84:45885/0 (socket says 172.21.15.84:45885) 2024-03-29T11:24:51.204 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:51.221+0000 7f069f7fe640 1 -- 172.21.15.84:0/1006711813 learned_addr learned my addr 172.21.15.84:0/1006711813 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-29T11:24:51.205 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:51.222+0000 7f069effd640 1 -- 172.21.15.84:0/1006711813 <== mon.0 v1:172.21.15.84:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 837388021 0 0) 0x7f06a006f2a0 con 0x7f06a01470b0 2024-03-29T11:24:51.205 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:51.222+0000 7f069effd640 10 monclient(hunting): _negotiate 2024-03-29T11:24:51.205 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:51.222+0000 7f069effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:51.205 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:51.222+0000 7f069effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:51.205 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:51.222+0000 7f069effd640 10 monclient(hunting): my global_id is 4434 2024-03-29T11:24:51.205 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:51.222+0000 7f069effd640 1 -- 172.21.15.84:0/1006711813 --> v1:172.21.15.84:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0688003520 con 0x7f06a01470b0 2024-03-29T11:24:51.205 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:51.222+0000 7f069effd640 1 -- 172.21.15.84:0/1006711813 <== mon.1 v1:172.21.15.179:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1346447483 0 0) 0x7f06a0146b90 con 0x7f06a01439b0 2024-03-29T11:24:51.205 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:51.222+0000 7f069effd640 10 monclient(hunting): _negotiate 2024-03-29T11:24:51.205 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:51.222+0000 7f069effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:51.205 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:51.222+0000 7f069effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:51.205 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:51.222+0000 7f069effd640 10 monclient(hunting): my global_id is 4387 2024-03-29T11:24:51.206 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:51.222+0000 7f069effd640 1 -- 172.21.15.84:0/1006711813 --> v1:172.21.15.179:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f06a006f2a0 con 0x7f06a01439b0 2024-03-29T11:24:51.208 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:51.223+0000 7f069effd640 1 -- 172.21.15.84:0/1006711813 <== mon.0 v1:172.21.15.84:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 965486207 0 0) 0x7f0688003520 con 0x7f06a01470b0 2024-03-29T11:24:51.208 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:51.223+0000 7f069effd640 1 -- 172.21.15.84:0/1006711813 >> v1:172.21.15.84:6790/0 conn(0x7f06a00b4330 legacy=0x7f06a00b4710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:51.208 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:51.223+0000 7f069effd640 1 -- 172.21.15.84:0/1006711813 >> v1:172.21.15.179:6789/0 conn(0x7f06a01439b0 legacy=0x7f06a0151d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:51.208 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:51.223+0000 7f069effd640 10 monclient: _finish_hunting 0 2024-03-29T11:24:51.208 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:51.223+0000 7f069effd640 1 monclient: found mon.noname-a 2024-03-29T11:24:51.208 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:51.223+0000 7f069effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:24:51.208 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:51.223+0000 7f069effd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.84:6789/0 2024-03-29T11:24:51.208 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:51.223+0000 7f069effd640 1 -- 172.21.15.84:0/1006711813 --> v1:172.21.15.84:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06a006e0c0 con 0x7f06a01470b0 2024-03-29T11:24:51.209 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:51.223+0000 7f069effd640 10 monclient: _finish_auth 0 2024-03-29T11:24:51.209 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:51.223+0000 7f069effd640 10 monclient: _check_auth_tickets 2024-03-29T11:24:51.209 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:51.223+0000 7f069effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:24:51.209 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:51.223+0000 7f069effd640 1 -- 172.21.15.84:0/1006711813 <== mon.0 v1:172.21.15.84:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4236868694 0 0) 0x7f0698001a10 con 0x7f06a01470b0 2024-03-29T11:24:51.209 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:51.223+0000 7f069effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:51.209 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:51.223+0000 7f069effd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:24:51.209 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:51.223+0000 7f069effd640 10 monclient: dump: 2024-03-29T11:24:51.209 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T11:24:51.209 INFO:teuthology.misc.health.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:24:51.209 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T11:18:53.864025+0000 2024-03-29T11:24:51.209 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T11:18:53.864025+0000 2024-03-29T11:24:51.209 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:51.209 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:24:51.209 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T11:24:51.209 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T11:24:51.210 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T11:24:51.210 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:24:51.210 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:51.223+0000 7f06a66d6640 5 monclient: authenticate success, global_id 4434 2024-03-29T11:24:51.210 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:51.223+0000 7f06a66d6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:24:51.210 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:51.226+0000 7f069effd640 1 -- 172.21.15.84:0/1006711813 <== mon.0 v1:172.21.15.84:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0698001c30 con 0x7f06a01470b0 2024-03-29T11:24:51.210 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:51.226+0000 7f069effd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:24:51.210 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:51.226+0000 7f069effd640 1 -- 172.21.15.84:0/1006711813 <== mon.0 v1:172.21.15.84:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4236868694 0 0) 0x7f0698001fc0 con 0x7f06a01470b0 2024-03-29T11:24:51.210 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:51.226+0000 7f069effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:51.211 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:51.226+0000 7f069effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:24:51.211 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:51.226+0000 7f069effd640 10 monclient: dump: 2024-03-29T11:24:51.211 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T11:24:51.211 INFO:teuthology.misc.health.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:24:51.211 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T11:18:53.864025+0000 2024-03-29T11:24:51.211 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T11:18:53.864025+0000 2024-03-29T11:24:51.211 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:51.211 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:24:51.211 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T11:24:51.211 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T11:24:51.211 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T11:24:51.211 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:24:51.211 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:51.227+0000 7f06a66d6640 10 monclient: get_monmap_and_config success 2024-03-29T11:24:51.211 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:51.227+0000 7f06a66d6640 4 set_mon_vals no callback set 2024-03-29T11:24:51.211 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:51.228+0000 7f06a66d6640 10 monclient: shutdown 2024-03-29T11:24:51.212 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:51.228+0000 7f06a66d6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:24:51.212 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:51.228+0000 7f06a66d6640 1 -- 172.21.15.84:0/1006711813 >> v1:172.21.15.84:6789/0 conn(0x7f06a01470b0 legacy=0x7f06a00b2590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:51.212 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:51.230+0000 7f06a66d6640 1 -- 172.21.15.84:0/1006711813 shutdown_connections 2024-03-29T11:24:51.212 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:51.230+0000 7f06a66d6640 1 -- 172.21.15.84:0/1006711813 >> 172.21.15.84:0/1006711813 conn(0x7f06a0066a00 msgr2=0x7f06a0066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:24:51.213 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:51.230+0000 7f06a66d6640 1 -- 172.21.15.84:0/1006711813 shutdown_connections 2024-03-29T11:24:51.213 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:51.230+0000 7f06a66d6640 1 -- 172.21.15.84:0/1006711813 wait complete. 2024-03-29T11:24:51.214 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:51.231+0000 7f06a66d6640 10 monclient: build_initial_monmap 2024-03-29T11:24:51.214 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:51.231+0000 7f06a66d6640 10 monclient: monmap: 2024-03-29T11:24:51.214 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-29T11:24:51.214 INFO:teuthology.misc.health.smithi084.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:24:51.214 INFO:teuthology.misc.health.smithi084.stderr:last_changed 0.000000 2024-03-29T11:24:51.214 INFO:teuthology.misc.health.smithi084.stderr:created 0.000000 2024-03-29T11:24:51.214 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-29T11:24:51.214 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:24:51.215 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-29T11:24:51.215 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.noname-b 2024-03-29T11:24:51.215 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-c 2024-03-29T11:24:51.215 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:24:51.215 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:51.231+0000 7f06a66d6640 1 librados: starting msgr at 2024-03-29T11:24:51.215 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:51.231+0000 7f06a66d6640 1 librados: starting objecter 2024-03-29T11:24:51.215 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:51.232+0000 7f06a66d6640 1 Processor -- start 2024-03-29T11:24:51.218 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:51.232+0000 7f06a66d6640 1 -- start start 2024-03-29T11:24:51.218 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:51.232+0000 7f06a66d6640 1 librados: setting wanted keys 2024-03-29T11:24:51.218 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:51.232+0000 7f06a66d6640 1 librados: calling monclient init 2024-03-29T11:24:51.218 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:51.232+0000 7f06a66d6640 10 monclient: init 2024-03-29T11:24:51.218 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:51.232+0000 7f06a66d6640 10 monclient: _reopen_session rank -1 2024-03-29T11:24:51.218 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:51.232+0000 7f06a66d6640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:24:51.218 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:51.233+0000 7f06a66d6640 10 monclient(hunting): picked mon.noname-a con 0x7f06a00b4330 addr v1:172.21.15.84:6789/0 2024-03-29T11:24:51.218 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:51.233+0000 7f06a66d6640 10 monclient(hunting): picked mon.noname-b con 0x7f06a01439b0 addr v1:172.21.15.179:6789/0 2024-03-29T11:24:51.218 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:51.233+0000 7f06a66d6640 10 monclient(hunting): picked mon.noname-c con 0x7f06a01470b0 addr v1:172.21.15.84:6790/0 2024-03-29T11:24:51.218 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:51.233+0000 7f06a66d6640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f06a0140f70 con 0x7f06a00b4330 2024-03-29T11:24:51.218 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:51.233+0000 7f06a66d6640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f06a01dbbe0 con 0x7f06a01439b0 2024-03-29T11:24:51.218 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:51.233+0000 7f06a66d6640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f06a0141670 con 0x7f06a01470b0 2024-03-29T11:24:51.219 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:51.233+0000 7f06a66d6640 10 monclient(hunting): _renew_subs 2024-03-29T11:24:51.219 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:51.233+0000 7f06a66d6640 10 monclient(hunting): authenticate will time out at 1152.527194s 2024-03-29T11:24:51.219 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:51.236+0000 7f069f7fe640 1 --1- >> v1:172.21.15.179:6789/0 conn(0x7f06a01439b0 0x7f06a0140150 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.179:6789/0 says I am v1:172.21.15.84:47195/0 (socket says 172.21.15.84:47195) 2024-03-29T11:24:51.219 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:51.236+0000 7f069f7fe640 1 -- 172.21.15.84:0/2713040265 learned_addr learned my addr 172.21.15.84:0/2713040265 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-29T11:24:51.221 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:51.237+0000 7f069cff9640 10 client.?.objecter ms_handle_connect 0x7f06a00b4330 2024-03-29T11:24:51.221 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:51.237+0000 7f069cff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:24:51.222 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:51.237+0000 7f069cff9640 10 client.?.objecter ms_handle_connect 0x7f06a01439b0 2024-03-29T11:24:51.222 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:51.237+0000 7f069cff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:24:51.222 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:51.239+0000 7f069cff9640 10 client.?.objecter ms_handle_connect 0x7f06a01470b0 2024-03-29T11:24:51.222 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:51.239+0000 7f069cff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:24:51.222 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:51.239+0000 7f069cff9640 1 -- 172.21.15.84:0/2713040265 <== mon.0 v1:172.21.15.84:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2659359415 0 0) 0x7f06a0140f70 con 0x7f06a00b4330 2024-03-29T11:24:51.222 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:51.239+0000 7f069cff9640 10 monclient(hunting): _negotiate 2024-03-29T11:24:51.222 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:51.239+0000 7f069cff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:51.222 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:51.239+0000 7f069cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:51.222 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:51.239+0000 7f069cff9640 10 monclient(hunting): my global_id is 4437 2024-03-29T11:24:51.223 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:51.239+0000 7f069cff9640 1 -- 172.21.15.84:0/2713040265 --> v1:172.21.15.84:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f068c004eb0 con 0x7f06a00b4330 2024-03-29T11:24:51.223 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:51.239+0000 7f069cff9640 1 -- 172.21.15.84:0/2713040265 <== mon.1 v1:172.21.15.179:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3062738614 0 0) 0x7f06a01dbbe0 con 0x7f06a01439b0 2024-03-29T11:24:51.223 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:51.239+0000 7f069cff9640 10 monclient(hunting): _negotiate 2024-03-29T11:24:51.223 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:51.239+0000 7f069cff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:51.223 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:51.239+0000 7f069cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:51.223 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:51.239+0000 7f069cff9640 10 monclient(hunting): my global_id is 4390 2024-03-29T11:24:51.223 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:51.239+0000 7f069cff9640 1 -- 172.21.15.84:0/2713040265 --> v1:172.21.15.179:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f06a0140f70 con 0x7f06a01439b0 2024-03-29T11:24:51.223 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:51.240+0000 7f069cff9640 1 -- 172.21.15.84:0/2713040265 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2052396265 0 0) 0x7f06a0141670 con 0x7f06a01470b0 2024-03-29T11:24:51.224 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:51.240+0000 7f069cff9640 10 monclient(hunting): _negotiate 2024-03-29T11:24:51.224 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:51.240+0000 7f069cff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:51.224 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:51.240+0000 7f069cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:51.224 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:51.240+0000 7f069cff9640 10 monclient(hunting): my global_id is 4433 2024-03-29T11:24:51.224 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:51.240+0000 7f069cff9640 1 -- 172.21.15.84:0/2713040265 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f06a01dbbe0 con 0x7f06a01470b0 2024-03-29T11:24:51.224 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:51.241+0000 7f069cff9640 1 -- 172.21.15.84:0/2713040265 <== mon.1 v1:172.21.15.179:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3357755284 0 0) 0x7f06a0140f70 con 0x7f06a01439b0 2024-03-29T11:24:51.224 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:51.241+0000 7f069cff9640 1 -- 172.21.15.84:0/2713040265 --> v1:172.21.15.179:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f06a0141670 con 0x7f06a01439b0 2024-03-29T11:24:51.224 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:51.241+0000 7f069cff9640 1 -- 172.21.15.84:0/2713040265 <== mon.1 v1:172.21.15.179:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4236868694 0 0) 0x7f0698001950 con 0x7f06a01439b0 2024-03-29T11:24:51.224 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:51.241+0000 7f069cff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:51.224 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:51.241+0000 7f069cff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:24:51.224 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:51.241+0000 7f069cff9640 10 monclient(hunting): dump: 2024-03-29T11:24:51.224 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T11:24:51.224 INFO:teuthology.misc.health.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:24:51.224 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T11:18:53.864025+0000 2024-03-29T11:24:51.224 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T11:18:53.864025+0000 2024-03-29T11:24:51.225 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:51.225 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:24:51.225 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T11:24:51.225 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T11:24:51.225 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T11:24:51.225 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:24:51.225 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:51.241+0000 7f069cff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:24:51.225 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:51.241+0000 7f069cff9640 1 -- 172.21.15.84:0/2713040265 <== mon.0 v1:172.21.15.84:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1214059308 0 0) 0x7f068c004eb0 con 0x7f06a00b4330 2024-03-29T11:24:51.225 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:51.241+0000 7f069cff9640 1 -- 172.21.15.84:0/2713040265 --> v1:172.21.15.84:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f068c006b80 con 0x7f06a00b4330 2024-03-29T11:24:51.225 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:51.241+0000 7f069cff9640 1 -- 172.21.15.84:0/2713040265 <== mon.0 v1:172.21.15.84:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4236868694 0 0) 0x7f0694001890 con 0x7f06a00b4330 2024-03-29T11:24:51.225 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:51.241+0000 7f069cff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:51.225 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:51.241+0000 7f069cff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:24:51.225 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:51.241+0000 7f069cff9640 10 monclient(hunting): dump: 2024-03-29T11:24:51.225 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T11:24:51.225 INFO:teuthology.misc.health.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:24:51.226 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T11:18:53.864025+0000 2024-03-29T11:24:51.226 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T11:18:53.864025+0000 2024-03-29T11:24:51.226 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:51.226 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:24:51.226 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T11:24:51.226 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T11:24:51.226 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T11:24:51.226 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:24:51.226 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:51.241+0000 7f069cff9640 1 -- 172.21.15.84:0/2713040265 <== mon.1 v1:172.21.15.179:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1753371980 0 0) 0x7f06a0141670 con 0x7f06a01439b0 2024-03-29T11:24:51.226 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:51.241+0000 7f069cff9640 1 -- 172.21.15.84:0/2713040265 >> v1:172.21.15.84:6790/0 conn(0x7f06a01470b0 legacy=0x7f06a0140860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:51.226 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:51.241+0000 7f069cff9640 1 -- 172.21.15.84:0/2713040265 >> v1:172.21.15.84:6789/0 conn(0x7f06a00b4330 legacy=0x7f06a01dec00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:51.226 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:51.241+0000 7f069cff9640 10 monclient: _finish_hunting 0 2024-03-29T11:24:51.227 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:51.241+0000 7f069cff9640 1 monclient: found mon.b 2024-03-29T11:24:51.227 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:51.241+0000 7f069cff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:24:51.227 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:51.241+0000 7f069cff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-29T11:24:51.227 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:51.241+0000 7f069cff9640 1 -- 172.21.15.84:0/2713040265 --> v1:172.21.15.179:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06a0141800 con 0x7f06a01439b0 2024-03-29T11:24:51.227 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:51.241+0000 7f069cff9640 10 monclient: _finish_auth 0 2024-03-29T11:24:51.227 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:51.241+0000 7f069cff9640 10 monclient: _check_auth_tickets 2024-03-29T11:24:51.227 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:51.241+0000 7f069cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:24:51.228 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:51.243+0000 7f06a66d6640 5 monclient: authenticate success, global_id 4390 2024-03-29T11:24:51.228 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:51.243+0000 7f06a66d6640 10 monclient: _renew_subs 2024-03-29T11:24:51.228 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:51.243+0000 7f06a66d6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-29T11:24:51.228 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:51.243+0000 7f06a66d6640 1 -- 172.21.15.84:0/2713040265 --> v1:172.21.15.179:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f06a01d1960 con 0x7f06a01439b0 2024-03-29T11:24:51.228 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:51.243+0000 7f06a66d6640 10 client.4390.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:24:51.228 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:51.243+0000 7f06a66d6640 10 monclient: _renew_subs 2024-03-29T11:24:51.228 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:51.243+0000 7f06a66d6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-29T11:24:51.228 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:51.243+0000 7f06a66d6640 1 -- 172.21.15.84:0/2713040265 --> v1:172.21.15.179:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f06a01db8a0 con 0x7f06a01439b0 2024-03-29T11:24:51.228 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:51.243+0000 7f06a66d6640 1 librados: init done 2024-03-29T11:24:51.228 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:51.245+0000 7f06a66d6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:24:51.229 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:51.245+0000 7f06a66d6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:24:51.229 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:51.245+0000 7f06a66d6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-29T11:24:51.229 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:51.245+0000 7f06a66d6640 1 -- 172.21.15.84:0/2713040265 --> v1:172.21.15.179:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f06a01dbd90 con 0x7f06a01439b0 2024-03-29T11:24:51.229 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:51.246+0000 7f069cff9640 1 -- 172.21.15.84:0/2713040265 <== mon.1 v1:172.21.15.179:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f06980014f0 con 0x7f06a01439b0 2024-03-29T11:24:51.229 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:51.246+0000 7f069cff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:24:51.229 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:51.246+0000 7f069e7fc640 4 set_mon_vals no callback set 2024-03-29T11:24:51.230 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:51.247+0000 7f069cff9640 1 -- 172.21.15.84:0/2713040265 <== mon.1 v1:172.21.15.179:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4236868694 0 0) 0x7f0698001950 con 0x7f06a01439b0 2024-03-29T11:24:51.230 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:51.247+0000 7f069cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:51.230 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:51.248+0000 7f069cff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:24:51.230 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:51.248+0000 7f069cff9640 10 monclient: dump: 2024-03-29T11:24:51.230 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T11:24:51.230 INFO:teuthology.misc.health.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:24:51.230 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T11:18:53.864025+0000 2024-03-29T11:24:51.230 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T11:18:53.864025+0000 2024-03-29T11:24:51.230 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:51.231 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:24:51.231 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T11:24:51.231 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T11:24:51.231 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T11:24:51.231 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:24:51.234 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:51.251+0000 7f069cff9640 1 -- 172.21.15.84:0/2713040265 <== mon.1 v1:172.21.15.179:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4033942909 0 0) 0x7f069801ac80 con 0x7f06a01439b0 2024-03-29T11:24:51.235 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:51.252+0000 7f069cff9640 1 -- 172.21.15.84:0/2713040265 <== mon.1 v1:172.21.15.179:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2808139587 0 0) 0x7f06980544f0 con 0x7f06a01439b0 2024-03-29T11:24:51.235 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:51.252+0000 7f069cff9640 10 client.4390.objecter ms_dispatch 0x7f06a0000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:24:51.235 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:51.252+0000 7f069cff9640 3 client.4390.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:24:51.235 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:51.252+0000 7f069cff9640 3 client.4390.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:24:51.235 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:51.252+0000 7f069cff9640 20 client.4390.objecter dump_active .. 0 homeless 2024-03-29T11:24:51.235 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:51.252+0000 7f069ffff640 10 monclient: get_auth_request con 0x7f068c06e5d0 auth_method 0 2024-03-29T11:24:51.235 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:51.253+0000 7f069cff9640 10 client.4390.objecter ms_handle_connect 0x7f068c06e5d0 2024-03-29T11:24:51.238 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:51.256+0000 7f069cff9640 1 -- 172.21.15.84:0/2713040265 <== mon.1 v1:172.21.15.179:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f0698050790 con 0x7f06a01439b0 2024-03-29T11:24:51.238 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:51.256+0000 7f069cff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:24:51.239 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:51.256+0000 7f069cff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:24:51.595 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:51.612+0000 7f06a66d6640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:24:51.595 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:51.612+0000 7f06a66d6640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:24:51.595 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:51.612+0000 7f06a66d6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-29T11:24:51.595 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:51.612+0000 7f06a66d6640 1 -- 172.21.15.84:0/2713040265 --> v1:172.21.15.179:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f06a01dbd90 con 0x7f06a01439b0 2024-03-29T11:24:51.596 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:51.613+0000 7f069cff9640 1 -- 172.21.15.84:0/2713040265 <== mon.1 v1:172.21.15.179:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4209477549) 0x7f0698050790 con 0x7f06a01439b0 2024-03-29T11:24:51.596 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:51.613+0000 7f069cff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:24:51.596 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:51.613+0000 7f069cff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:24:51.599 INFO:teuthology.misc.health.smithi084.stdout: 2024-03-29T11:24:51.599 INFO:teuthology.misc.health.smithi084.stdout:{"epoch":10,"fsid":"2c3e002f-9eb0-4860-853b-4ef1ef562213","created":"2024-03-29T11:19:32.029169+0000","modified":"2024-03-29T11:19:34.200299+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:34.200299+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"5c39e276-2034-4716-bbc0-57a634eefa92","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"cd580e99-109f-4bf1-88c0-a5987bc8b9ed","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"11e60a5a-051d-453f-a190-66387cb35592","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"2c329fbe-bbc4-4030-bf87-d711cad04679","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f4213d4a-bd36-4433-91ae-d7e2b02481d3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"f0cf44ea-6055-48af-805b-aed858102ff9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"d895eaa0-cc0b-4ee4-9b5a-2dfcd89a60b7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"9bc5a7bc-8f52-4db9-96d6-de5e68d52eaa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:24:51.602 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:51.620+0000 7f06a66d6640 10 librados: watch_flush enter 2024-03-29T11:24:51.602 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:51.620+0000 7f06a66d6640 10 librados: watch_flush exit 2024-03-29T11:24:51.602 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:51.620+0000 7f06a66d6640 20 client.4390.objecter shutdown clearing up homeless session... 2024-03-29T11:24:51.602 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:51.620+0000 7f06a66d6640 10 client.4390.objecter successfully canceled tick 2024-03-29T11:24:51.604 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:51.622+0000 7f06a66d6640 1 -- 172.21.15.84:0/2713040265 >> v1:172.21.15.179:6816/2981134980 conn(0x7f068c06e5d0 legacy=0x7f068c070a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:51.605 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:51.622+0000 7f06a66d6640 10 monclient: shutdown 2024-03-29T11:24:51.605 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:51.622+0000 7f06a66d6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:24:51.605 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:51.622+0000 7f06a66d6640 1 -- 172.21.15.84:0/2713040265 >> v1:172.21.15.179:6789/0 conn(0x7f06a01439b0 legacy=0x7f06a0140150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:51.605 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:51.622+0000 7f06a66d6640 1 -- 172.21.15.84:0/2713040265 shutdown_connections 2024-03-29T11:24:51.605 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:51.622+0000 7f06a66d6640 1 -- 172.21.15.84:0/2713040265 >> 172.21.15.84:0/2713040265 conn(0x7f06a0066a00 msgr2=0x7f06a00b71b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:24:51.605 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:51.622+0000 7f06a66d6640 1 -- 172.21.15.84:0/2713040265 shutdown_connections 2024-03-29T11:24:51.605 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:51.622+0000 7f06a66d6640 1 -- 172.21.15.84:0/2713040265 wait complete. 2024-03-29T11:24:51.605 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:51.622+0000 7f06a66d6640 1 librados: shutdown 2024-03-29T11:24:51.624 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:24:58.433 DEBUG:teuthology.orchestra.run.smithi084:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:24:58.597 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:58.611+0000 7f36db59e640 10 monclient: get_monmap_and_config 2024-03-29T11:24:58.598 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:58.611+0000 7f36db59e640 10 monclient: build_initial_monmap 2024-03-29T11:24:58.598 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:58.611+0000 7f36db59e640 10 monclient: monmap: 2024-03-29T11:24:58.598 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-29T11:24:58.598 INFO:teuthology.misc.health.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:24:58.598 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T11:24:58.612288+0000 2024-03-29T11:24:58.598 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T11:24:58.612288+0000 2024-03-29T11:24:58.598 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-29T11:24:58.598 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:24:58.598 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-29T11:24:58.598 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.noname-c 2024-03-29T11:24:58.598 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-b 2024-03-29T11:24:58.598 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:24:58.599 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:58.614+0000 7f36db59e640 1 Processor -- start 2024-03-29T11:24:58.602 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:58.617+0000 7f36db59e640 1 -- start start 2024-03-29T11:24:58.603 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:58.617+0000 7f36db59e640 10 monclient: init 2024-03-29T11:24:58.603 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:58.618+0000 7f36db59e640 10 monclient: _reopen_session rank -1 2024-03-29T11:24:58.603 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:58.618+0000 7f36db59e640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:24:58.603 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:58.618+0000 7f36db59e640 10 monclient(hunting): picked mon.noname-a con 0x7f36d4029eb0 addr v1:172.21.15.84:6789/0 2024-03-29T11:24:58.603 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:58.618+0000 7f36db59e640 10 monclient(hunting): picked mon.noname-b con 0x7f36d402b0a0 addr v1:172.21.15.84:6790/0 2024-03-29T11:24:58.605 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:58.621+0000 7f36db59e640 10 monclient(hunting): picked mon.noname-c con 0x7f36d40c9570 addr v1:172.21.15.179:6789/0 2024-03-29T11:24:58.605 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:58.621+0000 7f36db59e640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f36d402a290 con 0x7f36d4029eb0 2024-03-29T11:24:58.605 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:58.621+0000 7f36db59e640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f36d40331c0 con 0x7f36d40c9570 2024-03-29T11:24:58.605 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:58.621+0000 7f36db59e640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f36d402c100 con 0x7f36d402b0a0 2024-03-29T11:24:58.605 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:58.621+0000 7f36db59e640 10 monclient(hunting): _renew_subs 2024-03-29T11:24:58.605 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:58.621+0000 7f36db59e640 10 monclient(hunting): authenticate will time out at 1159.916071s 2024-03-29T11:24:58.605 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:58.622+0000 7f36dad9d640 1 --1- >> v1:172.21.15.179:6789/0 conn(0x7f36d40c9570 0x7f36d40d74c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.179:6789/0 says I am v1:172.21.15.84:48497/0 (socket says 172.21.15.84:48497) 2024-03-29T11:24:58.605 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:58.622+0000 7f36dad9d640 1 -- 172.21.15.84:0/283326456 learned_addr learned my addr 172.21.15.84:0/283326456 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-29T11:24:58.606 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:58.622+0000 7f36d959a640 1 -- 172.21.15.84:0/283326456 <== mon.1 v1:172.21.15.179:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 213962477 0 0) 0x7f36d40331c0 con 0x7f36d40c9570 2024-03-29T11:24:58.607 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:58.622+0000 7f36d959a640 10 monclient(hunting): _negotiate 2024-03-29T11:24:58.607 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:58.622+0000 7f36d959a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:58.607 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:58.622+0000 7f36d959a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:58.607 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:58.622+0000 7f36d959a640 10 monclient(hunting): my global_id is 4393 2024-03-29T11:24:58.607 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:58.622+0000 7f36d959a640 1 -- 172.21.15.84:0/283326456 --> v1:172.21.15.179:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f36c4003520 con 0x7f36d40c9570 2024-03-29T11:24:58.607 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:58.623+0000 7f36d959a640 1 -- 172.21.15.84:0/283326456 <== mon.1 v1:172.21.15.179:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3545940642 0 0) 0x7f36c4003520 con 0x7f36d40c9570 2024-03-29T11:24:58.607 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:58.623+0000 7f36d959a640 1 -- 172.21.15.84:0/283326456 >> v1:172.21.15.84:6790/0 conn(0x7f36d402b0a0 legacy=0x7f36d40c8e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:24:58.607 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:58.623+0000 7f36d959a640 1 -- 172.21.15.84:0/283326456 >> v1:172.21.15.84:6789/0 conn(0x7f36d4029eb0 legacy=0x7f36d40cf120 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:24:58.608 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:58.625+0000 7f36d959a640 10 monclient: _finish_hunting 0 2024-03-29T11:24:58.608 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:58.625+0000 7f36d959a640 1 monclient: found mon.noname-c 2024-03-29T11:24:58.608 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:58.625+0000 7f36d959a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:24:58.608 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:58.625+0000 7f36d959a640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.179:6789/0 2024-03-29T11:24:58.608 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:58.625+0000 7f36d959a640 1 -- 172.21.15.84:0/283326456 --> v1:172.21.15.179:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36d402a440 con 0x7f36d40c9570 2024-03-29T11:24:58.608 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:58.625+0000 7f36d959a640 10 monclient: _finish_auth 0 2024-03-29T11:24:58.608 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:58.625+0000 7f36d959a640 10 monclient: _check_auth_tickets 2024-03-29T11:24:58.608 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:58.625+0000 7f36d959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:24:58.608 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:58.625+0000 7f36d959a640 1 -- 172.21.15.84:0/283326456 <== mon.1 v1:172.21.15.179:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4236868694 0 0) 0x7f36dc04f730 con 0x7f36d40c9570 2024-03-29T11:24:58.608 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:58.625+0000 7f36d959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:58.608 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:58.625+0000 7f36d959a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:24:58.608 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:58.625+0000 7f36d959a640 10 monclient: dump: 2024-03-29T11:24:58.608 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T11:24:58.608 INFO:teuthology.misc.health.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:24:58.609 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T11:18:53.864025+0000 2024-03-29T11:24:58.609 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T11:18:53.864025+0000 2024-03-29T11:24:58.609 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:58.609 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:24:58.609 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T11:24:58.609 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T11:24:58.609 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T11:24:58.609 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:24:58.609 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:58.625+0000 7f36db59e640 5 monclient: authenticate success, global_id 4393 2024-03-29T11:24:58.609 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:58.625+0000 7f36db59e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:24:58.610 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:58.625+0000 7f36d959a640 1 -- 172.21.15.84:0/283326456 <== mon.1 v1:172.21.15.179:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f36dc075f10 con 0x7f36d40c9570 2024-03-29T11:24:58.610 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:58.625+0000 7f36d959a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:24:58.611 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:58.625+0000 7f36d959a640 1 -- 172.21.15.84:0/283326456 <== mon.1 v1:172.21.15.179:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4236868694 0 0) 0x7f36dc0758b0 con 0x7f36d40c9570 2024-03-29T11:24:58.611 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:58.625+0000 7f36d959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:58.611 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:58.625+0000 7f36d959a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:24:58.611 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:58.625+0000 7f36d959a640 10 monclient: dump: 2024-03-29T11:24:58.611 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T11:24:58.611 INFO:teuthology.misc.health.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:24:58.611 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T11:18:53.864025+0000 2024-03-29T11:24:58.612 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T11:18:53.864025+0000 2024-03-29T11:24:58.612 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:58.612 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:24:58.612 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T11:24:58.612 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T11:24:58.612 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T11:24:58.612 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:24:58.612 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:58.625+0000 7f36db59e640 10 monclient: get_monmap_and_config success 2024-03-29T11:24:58.612 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:58.625+0000 7f36db59e640 4 set_mon_vals no callback set 2024-03-29T11:24:58.612 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:58.626+0000 7f36db59e640 10 monclient: shutdown 2024-03-29T11:24:58.612 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:58.626+0000 7f36db59e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:24:58.612 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:58.626+0000 7f36db59e640 1 -- 172.21.15.84:0/283326456 >> v1:172.21.15.179:6789/0 conn(0x7f36d40c9570 legacy=0x7f36d40d74c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:58.612 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:58.627+0000 7f36db59e640 1 -- 172.21.15.84:0/283326456 shutdown_connections 2024-03-29T11:24:58.612 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:58.627+0000 7f36db59e640 1 -- 172.21.15.84:0/283326456 >> 172.21.15.84:0/283326456 conn(0x7f36d40058f0 msgr2=0x7f36d4005d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:24:58.614 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:58.628+0000 7f36db59e640 1 -- 172.21.15.84:0/283326456 shutdown_connections 2024-03-29T11:24:58.614 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:58.628+0000 7f36db59e640 1 -- 172.21.15.84:0/283326456 wait complete. 2024-03-29T11:24:58.614 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:58.629+0000 7f36db59e640 10 monclient: build_initial_monmap 2024-03-29T11:24:58.614 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:58.629+0000 7f36db59e640 10 monclient: monmap: 2024-03-29T11:24:58.614 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-29T11:24:58.615 INFO:teuthology.misc.health.smithi084.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:24:58.615 INFO:teuthology.misc.health.smithi084.stderr:last_changed 0.000000 2024-03-29T11:24:58.615 INFO:teuthology.misc.health.smithi084.stderr:created 0.000000 2024-03-29T11:24:58.615 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-29T11:24:58.615 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:24:58.615 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-29T11:24:58.615 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.noname-b 2024-03-29T11:24:58.615 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-c 2024-03-29T11:24:58.615 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:24:58.615 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:58.629+0000 7f36db59e640 1 librados: starting msgr at 2024-03-29T11:24:58.615 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:58.629+0000 7f36db59e640 1 librados: starting objecter 2024-03-29T11:24:58.615 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:58.629+0000 7f36db59e640 1 Processor -- start 2024-03-29T11:24:58.615 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:58.629+0000 7f36db59e640 1 -- start start 2024-03-29T11:24:58.616 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:58.629+0000 7f36db59e640 1 librados: setting wanted keys 2024-03-29T11:24:58.616 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:58.629+0000 7f36db59e640 1 librados: calling monclient init 2024-03-29T11:24:58.616 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:58.629+0000 7f36db59e640 10 monclient: init 2024-03-29T11:24:58.616 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:58.630+0000 7f36db59e640 10 monclient: _reopen_session rank -1 2024-03-29T11:24:58.616 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:58.630+0000 7f36db59e640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:24:58.616 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:58.630+0000 7f36db59e640 10 monclient(hunting): picked mon.noname-a con 0x7f36d4029eb0 addr v1:172.21.15.84:6789/0 2024-03-29T11:24:58.616 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:58.632+0000 7f36db59e640 10 monclient(hunting): picked mon.noname-c con 0x7f36d402b0a0 addr v1:172.21.15.84:6790/0 2024-03-29T11:24:58.617 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:58.632+0000 7f36db59e640 10 monclient(hunting): picked mon.noname-b con 0x7f36d40c9570 addr v1:172.21.15.179:6789/0 2024-03-29T11:24:58.617 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:58.632+0000 7f36db59e640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f36d4160e00 con 0x7f36d4029eb0 2024-03-29T11:24:58.617 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:58.632+0000 7f36db59e640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f36d40317c0 con 0x7f36d40c9570 2024-03-29T11:24:58.617 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:58.633+0000 7f36db59e640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f36d416d110 con 0x7f36d402b0a0 2024-03-29T11:24:58.617 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:58.633+0000 7f36db59e640 10 monclient(hunting): _renew_subs 2024-03-29T11:24:58.617 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:58.633+0000 7f36db59e640 10 monclient(hunting): authenticate will time out at 1159.927122s 2024-03-29T11:24:58.619 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:58.636+0000 7f36dad9d640 1 --1- >> v1:172.21.15.179:6789/0 conn(0x7f36d40c9570 0x7f36d41606f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.179:6789/0 says I am v1:172.21.15.84:48631/0 (socket says 172.21.15.84:48631) 2024-03-29T11:24:58.619 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:58.636+0000 7f36dad9d640 1 -- 172.21.15.84:0/2827487751 learned_addr learned my addr 172.21.15.84:0/2827487751 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-29T11:24:58.619 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:58.636+0000 7f36caffd640 10 client.?.objecter ms_handle_connect 0x7f36d40c9570 2024-03-29T11:24:58.619 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:58.636+0000 7f36caffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:24:58.623 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:58.637+0000 7f36caffd640 1 -- 172.21.15.84:0/2827487751 <== mon.1 v1:172.21.15.179:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1545215271 0 0) 0x7f36d40317c0 con 0x7f36d40c9570 2024-03-29T11:24:58.623 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:58.637+0000 7f36caffd640 10 monclient(hunting): _negotiate 2024-03-29T11:24:58.623 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:58.637+0000 7f36caffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:58.623 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:58.637+0000 7f36caffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:58.623 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:58.637+0000 7f36caffd640 10 monclient(hunting): my global_id is 4396 2024-03-29T11:24:58.623 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:58.637+0000 7f36caffd640 1 -- 172.21.15.84:0/2827487751 --> v1:172.21.15.179:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f36bc004eb0 con 0x7f36d40c9570 2024-03-29T11:24:58.623 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:58.637+0000 7f36caffd640 1 -- 172.21.15.84:0/2827487751 <== mon.1 v1:172.21.15.179:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1465053691 0 0) 0x7f36bc004eb0 con 0x7f36d40c9570 2024-03-29T11:24:58.623 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:58.637+0000 7f36caffd640 1 -- 172.21.15.84:0/2827487751 --> v1:172.21.15.179:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f36d40317c0 con 0x7f36d40c9570 2024-03-29T11:24:58.623 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:58.637+0000 7f36caffd640 1 -- 172.21.15.84:0/2827487751 <== mon.1 v1:172.21.15.179:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4236868694 0 0) 0x7f36dc04f400 con 0x7f36d40c9570 2024-03-29T11:24:58.623 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:58.637+0000 7f36caffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:58.623 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:58.637+0000 7f36caffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:24:58.624 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:58.637+0000 7f36caffd640 10 monclient(hunting): dump: 2024-03-29T11:24:58.624 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T11:24:58.624 INFO:teuthology.misc.health.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:24:58.624 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T11:18:53.864025+0000 2024-03-29T11:24:58.624 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T11:18:53.864025+0000 2024-03-29T11:24:58.624 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:58.624 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:24:58.624 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T11:24:58.624 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T11:24:58.624 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T11:24:58.624 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:24:58.624 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:58.637+0000 7f36caffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:24:58.624 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:58.637+0000 7f36caffd640 10 client.?.objecter ms_handle_connect 0x7f36d402b0a0 2024-03-29T11:24:58.624 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:58.637+0000 7f36caffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:24:58.624 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:58.637+0000 7f36caffd640 1 -- 172.21.15.84:0/2827487751 <== mon.1 v1:172.21.15.179:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3807840382 0 0) 0x7f36d40317c0 con 0x7f36d40c9570 2024-03-29T11:24:58.625 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:58.638+0000 7f36caffd640 1 -- 172.21.15.84:0/2827487751 >> v1:172.21.15.84:6790/0 conn(0x7f36d402b0a0 legacy=0x7f36d415ffe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:58.625 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:58.638+0000 7f36caffd640 1 -- 172.21.15.84:0/2827487751 >> v1:172.21.15.84:6789/0 conn(0x7f36d4029eb0 legacy=0x7f36d416ca00 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:24:58.625 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:58.638+0000 7f36caffd640 10 monclient: _finish_hunting 0 2024-03-29T11:24:58.625 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:58.638+0000 7f36caffd640 1 monclient: found mon.b 2024-03-29T11:24:58.625 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:58.638+0000 7f36caffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:24:58.625 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:58.638+0000 7f36caffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-29T11:24:58.625 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:58.638+0000 7f36caffd640 1 -- 172.21.15.84:0/2827487751 --> v1:172.21.15.179:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36d416d2a0 con 0x7f36d40c9570 2024-03-29T11:24:58.625 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:58.638+0000 7f36caffd640 10 monclient: _finish_auth 0 2024-03-29T11:24:58.625 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:58.638+0000 7f36caffd640 10 monclient: _check_auth_tickets 2024-03-29T11:24:58.625 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:58.638+0000 7f36caffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:24:58.625 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:58.638+0000 7f36db59e640 5 monclient: authenticate success, global_id 4396 2024-03-29T11:24:58.625 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:58.638+0000 7f36db59e640 10 monclient: _renew_subs 2024-03-29T11:24:58.625 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:58.638+0000 7f36db59e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-29T11:24:58.625 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:58.638+0000 7f36db59e640 1 -- 172.21.15.84:0/2827487751 --> v1:172.21.15.179:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f36d4031cc0 con 0x7f36d40c9570 2024-03-29T11:24:58.625 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:58.638+0000 7f36db59e640 10 client.4396.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:24:58.626 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:58.638+0000 7f36db59e640 10 monclient: _renew_subs 2024-03-29T11:24:58.626 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:58.638+0000 7f36db59e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-29T11:24:58.626 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:58.638+0000 7f36db59e640 1 -- 172.21.15.84:0/2827487751 --> v1:172.21.15.179:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f36d4031f70 con 0x7f36d40c9570 2024-03-29T11:24:58.626 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:58.638+0000 7f36db59e640 1 librados: init done 2024-03-29T11:24:58.626 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:58.638+0000 7f36caffd640 1 -- 172.21.15.84:0/2827487751 <== mon.1 v1:172.21.15.179:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f36dc04f730 con 0x7f36d40c9570 2024-03-29T11:24:58.626 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:58.638+0000 7f36caffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:24:58.626 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:58.638+0000 7f36caffd640 1 -- 172.21.15.84:0/2827487751 <== mon.1 v1:172.21.15.179:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4236868694 0 0) 0x7f36dc067d90 con 0x7f36d40c9570 2024-03-29T11:24:58.626 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:58.638+0000 7f36caffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:58.626 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:58.638+0000 7f36caffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:24:58.626 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:58.638+0000 7f36caffd640 10 monclient: dump: 2024-03-29T11:24:58.626 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T11:24:58.626 INFO:teuthology.misc.health.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:24:58.626 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T11:18:53.864025+0000 2024-03-29T11:24:58.626 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T11:18:53.864025+0000 2024-03-29T11:24:58.626 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:58.627 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:24:58.627 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T11:24:58.627 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T11:24:58.627 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T11:24:58.627 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:24:58.627 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:58.638+0000 7f36cbfff640 4 set_mon_vals no callback set 2024-03-29T11:24:58.628 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:58.639+0000 7f36c8ff9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:24:58.628 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:58.640+0000 7f36c8ff9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:24:58.628 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:58.640+0000 7f36c8ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-29T11:24:58.628 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:58.640+0000 7f36c8ff9640 1 -- 172.21.15.84:0/2827487751 --> v1:172.21.15.179:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f36b4005080 con 0x7f36d40c9570 2024-03-29T11:24:58.628 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:58.640+0000 7f36caffd640 1 -- 172.21.15.84:0/2827487751 <== mon.1 v1:172.21.15.179:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4033942909 0 0) 0x7f36dc067ef0 con 0x7f36d40c9570 2024-03-29T11:24:58.628 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:58.641+0000 7f36caffd640 1 -- 172.21.15.84:0/2827487751 <== mon.1 v1:172.21.15.179:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2808139587 0 0) 0x7f36dc0ff7a0 con 0x7f36d40c9570 2024-03-29T11:24:58.628 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:58.641+0000 7f36caffd640 10 client.4396.objecter ms_dispatch 0x7f36dc000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:24:58.628 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:58.641+0000 7f36caffd640 3 client.4396.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:24:58.629 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:58.641+0000 7f36caffd640 3 client.4396.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:24:58.629 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:58.641+0000 7f36caffd640 20 client.4396.objecter dump_active .. 0 homeless 2024-03-29T11:24:58.629 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:58.641+0000 7f36da59c640 10 monclient: get_auth_request con 0x7f36bc06dad0 auth_method 0 2024-03-29T11:24:58.629 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:58.642+0000 7f36caffd640 10 client.4396.objecter ms_handle_connect 0x7f36bc06dad0 2024-03-29T11:24:58.632 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:58.648+0000 7f36caffd640 1 -- 172.21.15.84:0/2827487751 <== mon.1 v1:172.21.15.179:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f36dc0cb080 con 0x7f36d40c9570 2024-03-29T11:24:58.633 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:58.648+0000 7f36caffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:24:58.633 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:58.648+0000 7f36caffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:24:58.975 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:58.991+0000 7f36c8ff9640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:24:58.975 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:58.991+0000 7f36c8ff9640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:24:58.975 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:58.992+0000 7f36c8ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-29T11:24:58.975 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:58.992+0000 7f36c8ff9640 1 -- 172.21.15.84:0/2827487751 --> v1:172.21.15.179:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f36b4005330 con 0x7f36d40c9570 2024-03-29T11:24:58.975 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:58.993+0000 7f36caffd640 1 -- 172.21.15.84:0/2827487751 <== mon.1 v1:172.21.15.179:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4209477549) 0x7f36dc068220 con 0x7f36d40c9570 2024-03-29T11:24:58.976 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:58.993+0000 7f36caffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:24:58.976 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:58.993+0000 7f36caffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:24:58.976 INFO:teuthology.misc.health.smithi084.stdout: 2024-03-29T11:24:58.976 INFO:teuthology.misc.health.smithi084.stdout:{"epoch":10,"fsid":"2c3e002f-9eb0-4860-853b-4ef1ef562213","created":"2024-03-29T11:19:32.029169+0000","modified":"2024-03-29T11:19:34.200299+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:34.200299+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"5c39e276-2034-4716-bbc0-57a634eefa92","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"cd580e99-109f-4bf1-88c0-a5987bc8b9ed","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"11e60a5a-051d-453f-a190-66387cb35592","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"2c329fbe-bbc4-4030-bf87-d711cad04679","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f4213d4a-bd36-4433-91ae-d7e2b02481d3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"f0cf44ea-6055-48af-805b-aed858102ff9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"d895eaa0-cc0b-4ee4-9b5a-2dfcd89a60b7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"9bc5a7bc-8f52-4db9-96d6-de5e68d52eaa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:24:58.979 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:58.997+0000 7f36c8ff9640 10 librados: watch_flush enter 2024-03-29T11:24:58.980 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:58.997+0000 7f36c8ff9640 10 librados: watch_flush exit 2024-03-29T11:24:58.980 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:58.997+0000 7f36c8ff9640 20 client.4396.objecter shutdown clearing up homeless session... 2024-03-29T11:24:58.981 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:58.997+0000 7f36c8ff9640 10 client.4396.objecter successfully canceled tick 2024-03-29T11:24:58.981 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:58.998+0000 7f36c8ff9640 1 -- 172.21.15.84:0/2827487751 >> v1:172.21.15.179:6816/2981134980 conn(0x7f36bc06dad0 legacy=0x7f36bc06ff90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:58.981 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:58.998+0000 7f36c8ff9640 10 monclient: shutdown 2024-03-29T11:24:58.982 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:58.998+0000 7f36c8ff9640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:24:58.982 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:58.998+0000 7f36c8ff9640 1 -- 172.21.15.84:0/2827487751 >> v1:172.21.15.179:6789/0 conn(0x7f36d40c9570 legacy=0x7f36d41606f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:58.982 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:58.999+0000 7f36c8ff9640 1 -- 172.21.15.84:0/2827487751 shutdown_connections 2024-03-29T11:24:58.983 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:58.999+0000 7f36c8ff9640 1 -- 172.21.15.84:0/2827487751 >> 172.21.15.84:0/2827487751 conn(0x7f36d40058f0 msgr2=0x7f36d40043d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:24:58.983 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:59.001+0000 7f36c8ff9640 1 -- 172.21.15.84:0/2827487751 shutdown_connections 2024-03-29T11:24:58.987 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:59.004+0000 7f36c8ff9640 1 -- 172.21.15.84:0/2827487751 wait complete. 2024-03-29T11:24:58.987 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:24:59.004+0000 7f36c8ff9640 1 librados: shutdown 2024-03-29T11:24:59.006 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:25:05.812 DEBUG:teuthology.orchestra.run.smithi084:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:25:05.955 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:05.972+0000 7fcc05f3d640 10 monclient: get_monmap_and_config 2024-03-29T11:25:05.956 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:05.972+0000 7fcc05f3d640 10 monclient: build_initial_monmap 2024-03-29T11:25:05.956 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:05.972+0000 7fcc05f3d640 10 monclient: monmap: 2024-03-29T11:25:05.956 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-29T11:25:05.956 INFO:teuthology.misc.health.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:25:05.956 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T11:25:05.973008+0000 2024-03-29T11:25:05.956 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T11:25:05.973008+0000 2024-03-29T11:25:05.957 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-29T11:25:05.957 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:25:05.957 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-29T11:25:05.957 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.noname-c 2024-03-29T11:25:05.957 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-b 2024-03-29T11:25:05.957 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:25:05.983 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:05.973+0000 7fcc05f3d640 1 Processor -- start 2024-03-29T11:25:05.984 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:05.973+0000 7fcc05f3d640 1 -- start start 2024-03-29T11:25:05.984 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:05.973+0000 7fcc05f3d640 10 monclient: init 2024-03-29T11:25:05.984 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:05.974+0000 7fcc05f3d640 10 monclient: _reopen_session rank -1 2024-03-29T11:25:05.984 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:05.974+0000 7fcc05f3d640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:25:05.984 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:05.974+0000 7fcc05f3d640 10 monclient(hunting): picked mon.noname-c con 0x7fcc001422b0 addr v1:172.21.15.179:6789/0 2024-03-29T11:25:05.984 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:05.974+0000 7fcc05f3d640 10 monclient(hunting): picked mon.noname-a con 0x7fcc00142690 addr v1:172.21.15.84:6789/0 2024-03-29T11:25:05.984 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:05.974+0000 7fcc05f3d640 10 monclient(hunting): picked mon.noname-b con 0x7fcc000b17d0 addr v1:172.21.15.84:6790/0 2024-03-29T11:25:05.984 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:05.974+0000 7fcc05f3d640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcc00142c80 con 0x7fcc00142690 2024-03-29T11:25:05.984 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:05.974+0000 7fcc05f3d640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcc000aae40 con 0x7fcc001422b0 2024-03-29T11:25:05.984 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:05.974+0000 7fcc05f3d640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcc0013fbd0 con 0x7fcc000b17d0 2024-03-29T11:25:05.984 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:05.974+0000 7fcc05f3d640 10 monclient(hunting): _renew_subs 2024-03-29T11:25:05.984 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:05.974+0000 7fcc05f3d640 10 monclient(hunting): authenticate will time out at 1167.268540s 2024-03-29T11:25:05.984 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:05.974+0000 7fcbfeffd640 1 --1- >> v1:172.21.15.84:6789/0 conn(0x7fcc00142690 0x7fcc000b2590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.84:6789/0 says I am v1:172.21.15.84:50738/0 (socket says 172.21.15.84:50738) 2024-03-29T11:25:05.985 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:05.974+0000 7fcbfffff640 1 --1- >> v1:172.21.15.84:6790/0 conn(0x7fcc000b17d0 0x7fcc000b2ca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.84:6790/0 says I am v1:172.21.15.84:45476/0 (socket says 172.21.15.84:45476) 2024-03-29T11:25:05.985 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:05.974+0000 7fcbfffff640 1 -- 172.21.15.84:0/2124827067 learned_addr learned my addr 172.21.15.84:0/2124827067 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-29T11:25:05.985 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:05.975+0000 7fcbfe7fc640 1 -- 172.21.15.84:0/2124827067 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2910933735 0 0) 0x7fcc0013fbd0 con 0x7fcc000b17d0 2024-03-29T11:25:05.985 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:05.975+0000 7fcbfe7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:25:05.985 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:05.975+0000 7fcbfe7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:25:05.985 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:05.975+0000 7fcbfe7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:25:05.985 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:05.975+0000 7fcbfe7fc640 10 monclient(hunting): my global_id is 4439 2024-03-29T11:25:05.985 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:05.975+0000 7fcbfe7fc640 1 -- 172.21.15.84:0/2124827067 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcbec003520 con 0x7fcc000b17d0 2024-03-29T11:25:05.985 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:05.975+0000 7fcbfe7fc640 1 -- 172.21.15.84:0/2124827067 <== mon.0 v1:172.21.15.84:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2572323143 0 0) 0x7fcc00142c80 con 0x7fcc00142690 2024-03-29T11:25:05.985 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:05.975+0000 7fcbfe7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:25:05.985 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:05.975+0000 7fcbfe7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:25:05.985 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:05.975+0000 7fcbfe7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:25:05.985 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:05.975+0000 7fcbfe7fc640 10 monclient(hunting): my global_id is 4440 2024-03-29T11:25:05.985 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:05.975+0000 7fcbfe7fc640 1 -- 172.21.15.84:0/2124827067 --> v1:172.21.15.84:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcc0013fbd0 con 0x7fcc00142690 2024-03-29T11:25:05.986 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:05.975+0000 7fcbfe7fc640 1 -- 172.21.15.84:0/2124827067 <== mon.1 v1:172.21.15.179:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3556223076 0 0) 0x7fcc000aae40 con 0x7fcc001422b0 2024-03-29T11:25:05.986 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:05.975+0000 7fcbfe7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:25:05.986 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:05.975+0000 7fcbfe7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:25:05.986 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:05.975+0000 7fcbfe7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:25:05.986 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:05.975+0000 7fcbfe7fc640 10 monclient(hunting): my global_id is 4399 2024-03-29T11:25:05.986 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:05.976+0000 7fcbfe7fc640 1 -- 172.21.15.84:0/2124827067 --> v1:172.21.15.179:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcc00142c80 con 0x7fcc001422b0 2024-03-29T11:25:05.986 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:05.976+0000 7fcbfe7fc640 1 -- 172.21.15.84:0/2124827067 <== mon.1 v1:172.21.15.179:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4033116892 0 0) 0x7fcc00142c80 con 0x7fcc001422b0 2024-03-29T11:25:05.986 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:05.976+0000 7fcbfe7fc640 1 -- 172.21.15.84:0/2124827067 >> v1:172.21.15.84:6790/0 conn(0x7fcc000b17d0 legacy=0x7fcc000b2ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:05.986 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:05.976+0000 7fcbfe7fc640 1 -- 172.21.15.84:0/2124827067 >> v1:172.21.15.84:6789/0 conn(0x7fcc00142690 legacy=0x7fcc000b2590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:05.986 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:05.976+0000 7fcbfe7fc640 10 monclient: _finish_hunting 0 2024-03-29T11:25:05.986 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:05.976+0000 7fcbfe7fc640 1 monclient: found mon.noname-c 2024-03-29T11:25:05.986 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:05.976+0000 7fcbfe7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:25:05.986 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:05.976+0000 7fcbfe7fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.179:6789/0 2024-03-29T11:25:05.986 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:05.976+0000 7fcbfe7fc640 1 -- 172.21.15.84:0/2124827067 --> v1:172.21.15.179:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc00151940 con 0x7fcc001422b0 2024-03-29T11:25:05.987 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:05.976+0000 7fcbfe7fc640 10 monclient: _finish_auth 0 2024-03-29T11:25:05.987 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:05.976+0000 7fcbfe7fc640 10 monclient: _check_auth_tickets 2024-03-29T11:25:05.987 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:05.976+0000 7fcbfe7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:25:05.987 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:05.976+0000 7fcbfe7fc640 1 -- 172.21.15.84:0/2124827067 <== mon.1 v1:172.21.15.179:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4236868694 0 0) 0x7fcbf00018f0 con 0x7fcc001422b0 2024-03-29T11:25:05.987 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:05.976+0000 7fcbfe7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:25:05.987 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:05.976+0000 7fcbfe7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:25:05.987 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:05.976+0000 7fcbfe7fc640 10 monclient: dump: 2024-03-29T11:25:05.987 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T11:25:05.987 INFO:teuthology.misc.health.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:25:05.987 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T11:18:53.864025+0000 2024-03-29T11:25:05.987 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T11:18:53.864025+0000 2024-03-29T11:25:05.987 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T11:25:05.987 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:25:05.987 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T11:25:05.988 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T11:25:05.988 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T11:25:05.988 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:25:05.988 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:05.976+0000 7fcc05f3d640 5 monclient: authenticate success, global_id 4399 2024-03-29T11:25:05.988 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:05.976+0000 7fcc05f3d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:25:05.988 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:05.976+0000 7fcbfe7fc640 1 -- 172.21.15.84:0/2124827067 <== mon.1 v1:172.21.15.179:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fcbf0001ad0 con 0x7fcc001422b0 2024-03-29T11:25:05.988 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:05.976+0000 7fcbfe7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:25:05.989 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:05.976+0000 7fcc05f3d640 10 monclient: get_monmap_and_config success 2024-03-29T11:25:05.989 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:05.976+0000 7fcc05f3d640 4 set_mon_vals no callback set 2024-03-29T11:25:05.989 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:05.976+0000 7fcbfe7fc640 1 -- 172.21.15.84:0/2124827067 <== mon.1 v1:172.21.15.179:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4236868694 0 0) 0x7fcbf00018f0 con 0x7fcc001422b0 2024-03-29T11:25:05.989 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:05.976+0000 7fcbfe7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:25:05.989 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:05.976+0000 7fcbfe7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:25:05.989 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:05.976+0000 7fcbfe7fc640 10 monclient: dump: 2024-03-29T11:25:05.990 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T11:25:05.990 INFO:teuthology.misc.health.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:25:05.990 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T11:18:53.864025+0000 2024-03-29T11:25:05.990 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T11:18:53.864025+0000 2024-03-29T11:25:05.990 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T11:25:05.990 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:25:05.990 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T11:25:05.990 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T11:25:05.990 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T11:25:05.990 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:25:05.990 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:05.978+0000 7fcc05f3d640 10 monclient: shutdown 2024-03-29T11:25:05.990 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:05.978+0000 7fcc05f3d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:25:05.990 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:05.978+0000 7fcc05f3d640 1 -- 172.21.15.84:0/2124827067 >> v1:172.21.15.179:6789/0 conn(0x7fcc001422b0 legacy=0x7fcc0013f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:05.991 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:05.978+0000 7fcc05f3d640 1 -- 172.21.15.84:0/2124827067 shutdown_connections 2024-03-29T11:25:05.991 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:05.978+0000 7fcc05f3d640 1 -- 172.21.15.84:0/2124827067 >> 172.21.15.84:0/2124827067 conn(0x7fcc00066a00 msgr2=0x7fcc00066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:25:05.991 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:05.978+0000 7fcc05f3d640 1 -- 172.21.15.84:0/2124827067 shutdown_connections 2024-03-29T11:25:05.991 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:05.978+0000 7fcc05f3d640 1 -- 172.21.15.84:0/2124827067 wait complete. 2024-03-29T11:25:05.991 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:05.979+0000 7fcc05f3d640 10 monclient: build_initial_monmap 2024-03-29T11:25:05.991 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:05.979+0000 7fcc05f3d640 10 monclient: monmap: 2024-03-29T11:25:05.991 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-29T11:25:05.991 INFO:teuthology.misc.health.smithi084.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:25:05.991 INFO:teuthology.misc.health.smithi084.stderr:last_changed 0.000000 2024-03-29T11:25:05.991 INFO:teuthology.misc.health.smithi084.stderr:created 0.000000 2024-03-29T11:25:05.991 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-29T11:25:05.991 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:25:05.991 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-29T11:25:05.991 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.noname-b 2024-03-29T11:25:05.991 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-c 2024-03-29T11:25:05.992 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:25:05.992 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:05.979+0000 7fcc05f3d640 1 librados: starting msgr at 2024-03-29T11:25:05.992 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:05.979+0000 7fcc05f3d640 1 librados: starting objecter 2024-03-29T11:25:05.992 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:05.979+0000 7fcc05f3d640 1 Processor -- start 2024-03-29T11:25:05.992 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:05.980+0000 7fcc05f3d640 1 -- start start 2024-03-29T11:25:05.992 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:05.980+0000 7fcc05f3d640 1 librados: setting wanted keys 2024-03-29T11:25:05.992 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:05.980+0000 7fcc05f3d640 1 librados: calling monclient init 2024-03-29T11:25:05.992 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:05.980+0000 7fcc05f3d640 10 monclient: init 2024-03-29T11:25:05.992 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:05.981+0000 7fcc05f3d640 10 monclient: _reopen_session rank -1 2024-03-29T11:25:05.992 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:05.981+0000 7fcc05f3d640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:25:05.992 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:05.981+0000 7fcc05f3d640 10 monclient(hunting): picked mon.noname-a con 0x7fcc000b17d0 addr v1:172.21.15.84:6789/0 2024-03-29T11:25:05.992 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:05.981+0000 7fcc05f3d640 10 monclient(hunting): picked mon.noname-c con 0x7fcc001422b0 addr v1:172.21.15.84:6790/0 2024-03-29T11:25:05.992 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:05.981+0000 7fcc05f3d640 10 monclient(hunting): picked mon.noname-b con 0x7fcc00142690 addr v1:172.21.15.179:6789/0 2024-03-29T11:25:05.992 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:05.981+0000 7fcc05f3d640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcc0006f120 con 0x7fcc000b17d0 2024-03-29T11:25:05.993 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:05.981+0000 7fcc05f3d640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcc0013ce00 con 0x7fcc00142690 2024-03-29T11:25:05.993 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:05.981+0000 7fcc05f3d640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcc000ace40 con 0x7fcc001422b0 2024-03-29T11:25:05.993 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:05.981+0000 7fcc05f3d640 10 monclient(hunting): _renew_subs 2024-03-29T11:25:05.993 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:05.981+0000 7fcc05f3d640 10 monclient(hunting): authenticate will time out at 1167.275250s 2024-03-29T11:25:05.993 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:05.981+0000 7fcbfffff640 1 --1- >> v1:172.21.15.179:6789/0 conn(0x7fcc00142690 0x7fcc001dda30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.179:6789/0 says I am v1:172.21.15.84:46394/0 (socket says 172.21.15.84:46394) 2024-03-29T11:25:05.993 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:05.981+0000 7fcbfffff640 1 -- 172.21.15.84:0/2127700912 learned_addr learned my addr 172.21.15.84:0/2127700912 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-29T11:25:05.993 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:05.981+0000 7fcbf7fff640 10 client.?.objecter ms_handle_connect 0x7fcc00142690 2024-03-29T11:25:05.993 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:05.981+0000 7fcbf7fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:25:05.993 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:05.981+0000 7fcbf7fff640 10 client.?.objecter ms_handle_connect 0x7fcc000b17d0 2024-03-29T11:25:05.993 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:05.981+0000 7fcbf7fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:25:05.993 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:05.982+0000 7fcbf7fff640 10 client.?.objecter ms_handle_connect 0x7fcc001422b0 2024-03-29T11:25:05.993 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:05.982+0000 7fcbf7fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:25:05.993 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:05.982+0000 7fcbf7fff640 1 -- 172.21.15.84:0/2127700912 <== mon.0 v1:172.21.15.84:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1328117422 0 0) 0x7fcc0006f120 con 0x7fcc000b17d0 2024-03-29T11:25:05.993 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:05.982+0000 7fcbf7fff640 10 monclient(hunting): _negotiate 2024-03-29T11:25:05.993 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:05.982+0000 7fcbf7fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:25:05.994 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:05.982+0000 7fcbf7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:25:05.994 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:05.982+0000 7fcbf7fff640 10 monclient(hunting): my global_id is 4443 2024-03-29T11:25:05.994 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:05.982+0000 7fcbf7fff640 1 -- 172.21.15.84:0/2127700912 --> v1:172.21.15.84:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcbe0004eb0 con 0x7fcc000b17d0 2024-03-29T11:25:05.994 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:05.982+0000 7fcbf7fff640 1 -- 172.21.15.84:0/2127700912 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2136100118 0 0) 0x7fcc000ace40 con 0x7fcc001422b0 2024-03-29T11:25:05.994 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:05.982+0000 7fcbf7fff640 10 monclient(hunting): _negotiate 2024-03-29T11:25:05.994 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:05.982+0000 7fcbf7fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:25:05.994 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:05.982+0000 7fcbf7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:25:05.994 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:05.982+0000 7fcbf7fff640 10 monclient(hunting): my global_id is 4442 2024-03-29T11:25:05.994 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:05.982+0000 7fcbf7fff640 1 -- 172.21.15.84:0/2127700912 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcbe00051d0 con 0x7fcc001422b0 2024-03-29T11:25:05.994 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:05.982+0000 7fcbf7fff640 1 -- 172.21.15.84:0/2127700912 <== mon.1 v1:172.21.15.179:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2936139938 0 0) 0x7fcc0013ce00 con 0x7fcc00142690 2024-03-29T11:25:05.994 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:05.982+0000 7fcbf7fff640 10 monclient(hunting): _negotiate 2024-03-29T11:25:05.994 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:05.982+0000 7fcbf7fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:25:05.994 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:05.982+0000 7fcbf7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:25:05.995 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:05.982+0000 7fcbf7fff640 10 monclient(hunting): my global_id is 4402 2024-03-29T11:25:05.995 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:05.982+0000 7fcbf7fff640 1 -- 172.21.15.84:0/2127700912 --> v1:172.21.15.179:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcbe0005510 con 0x7fcc00142690 2024-03-29T11:25:05.995 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:05.983+0000 7fcbf7fff640 1 -- 172.21.15.84:0/2127700912 <== mon.0 v1:172.21.15.84:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 721846251 0 0) 0x7fcbe0004eb0 con 0x7fcc000b17d0 2024-03-29T11:25:05.995 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:05.983+0000 7fcbf7fff640 1 -- 172.21.15.84:0/2127700912 --> v1:172.21.15.84:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fcc0013ce00 con 0x7fcc000b17d0 2024-03-29T11:25:05.995 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:05.983+0000 7fcbf7fff640 1 -- 172.21.15.84:0/2127700912 <== mon.0 v1:172.21.15.84:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4236868694 0 0) 0x7fcbf00018f0 con 0x7fcc000b17d0 2024-03-29T11:25:05.995 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:05.983+0000 7fcbf7fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:25:05.995 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:05.983+0000 7fcbf7fff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:25:05.995 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:05.983+0000 7fcbf7fff640 10 monclient(hunting): dump: 2024-03-29T11:25:05.995 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T11:25:05.995 INFO:teuthology.misc.health.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:25:05.995 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T11:18:53.864025+0000 2024-03-29T11:25:05.995 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T11:18:53.864025+0000 2024-03-29T11:25:05.995 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T11:25:05.995 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:25:05.995 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T11:25:05.996 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T11:25:05.996 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T11:25:05.996 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:25:05.996 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:05.983+0000 7fcbf7fff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:25:05.996 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:05.983+0000 7fcbf7fff640 1 -- 172.21.15.84:0/2127700912 <== mon.2 v1:172.21.15.84:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2779038585 0 0) 0x7fcbe00051d0 con 0x7fcc001422b0 2024-03-29T11:25:05.996 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:05.983+0000 7fcbf7fff640 1 -- 172.21.15.84:0/2127700912 --> v1:172.21.15.84:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fcbe0004eb0 con 0x7fcc001422b0 2024-03-29T11:25:05.996 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:05.983+0000 7fcbf7fff640 1 -- 172.21.15.84:0/2127700912 <== mon.2 v1:172.21.15.84:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4236868694 0 0) 0x7fcbe8001790 con 0x7fcc001422b0 2024-03-29T11:25:05.996 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:05.983+0000 7fcbf7fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:25:05.996 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:05.983+0000 7fcbf7fff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:25:05.996 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:05.983+0000 7fcbf7fff640 10 monclient(hunting): dump: 2024-03-29T11:25:05.996 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T11:25:05.996 INFO:teuthology.misc.health.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:25:05.996 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T11:18:53.864025+0000 2024-03-29T11:25:05.996 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T11:18:53.864025+0000 2024-03-29T11:25:05.997 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T11:25:05.997 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:25:05.997 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T11:25:05.997 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T11:25:05.997 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T11:25:05.997 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:25:05.997 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:05.984+0000 7fcbf7fff640 1 -- 172.21.15.84:0/2127700912 <== mon.1 v1:172.21.15.179:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 103495603 0 0) 0x7fcbe0005510 con 0x7fcc00142690 2024-03-29T11:25:05.997 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:05.984+0000 7fcbf7fff640 1 -- 172.21.15.84:0/2127700912 --> v1:172.21.15.179:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fcbe0006380 con 0x7fcc00142690 2024-03-29T11:25:05.997 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:05.984+0000 7fcbf7fff640 1 -- 172.21.15.84:0/2127700912 <== mon.1 v1:172.21.15.179:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4236868694 0 0) 0x7fcbf80018d0 con 0x7fcc00142690 2024-03-29T11:25:05.997 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:05.984+0000 7fcbf7fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:25:05.997 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:05.984+0000 7fcbf7fff640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:25:05.997 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:05.984+0000 7fcbf7fff640 10 monclient(hunting): dump: 2024-03-29T11:25:05.997 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T11:25:05.997 INFO:teuthology.misc.health.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:25:05.998 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T11:18:53.864025+0000 2024-03-29T11:25:05.998 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T11:18:53.864025+0000 2024-03-29T11:25:05.998 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T11:25:05.998 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:25:05.998 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T11:25:05.998 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T11:25:05.998 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T11:25:05.998 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:25:05.998 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:05.984+0000 7fcbf7fff640 1 -- 172.21.15.84:0/2127700912 <== mon.0 v1:172.21.15.84:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3353022318 0 0) 0x7fcc0013ce00 con 0x7fcc000b17d0 2024-03-29T11:25:05.998 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:05.984+0000 7fcbf7fff640 1 -- 172.21.15.84:0/2127700912 >> v1:172.21.15.84:6790/0 conn(0x7fcc001422b0 legacy=0x7fcc0006fcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:05.998 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:05.984+0000 7fcbf7fff640 1 -- 172.21.15.84:0/2127700912 >> v1:172.21.15.179:6789/0 conn(0x7fcc00142690 legacy=0x7fcc001dda30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:05.998 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:05.984+0000 7fcbf7fff640 10 monclient: _finish_hunting 0 2024-03-29T11:25:05.998 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:05.984+0000 7fcbf7fff640 1 monclient: found mon.a 2024-03-29T11:25:05.998 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:05.984+0000 7fcbf7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:25:05.999 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:05.984+0000 7fcbf7fff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.84:6789/0 2024-03-29T11:25:05.999 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:05.984+0000 7fcbf7fff640 1 -- 172.21.15.84:0/2127700912 --> v1:172.21.15.84:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc000ad120 con 0x7fcc000b17d0 2024-03-29T11:25:05.999 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:05.984+0000 7fcbf7fff640 10 monclient: _finish_auth 0 2024-03-29T11:25:05.999 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:05.984+0000 7fcbf7fff640 10 monclient: _check_auth_tickets 2024-03-29T11:25:05.999 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:05.984+0000 7fcbf7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:25:05.999 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:05.984+0000 7fcc05f3d640 5 monclient: authenticate success, global_id 4443 2024-03-29T11:25:05.999 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:05.984+0000 7fcc05f3d640 10 monclient: _renew_subs 2024-03-29T11:25:05.999 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:05.984+0000 7fcc05f3d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.84:6789/0 2024-03-29T11:25:05.999 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:05.984+0000 7fcc05f3d640 1 -- 172.21.15.84:0/2127700912 --> v1:172.21.15.84:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcc000ad400 con 0x7fcc000b17d0 2024-03-29T11:25:05.999 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:05.984+0000 7fcc05f3d640 10 client.4443.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:25:05.999 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:05.984+0000 7fcc05f3d640 10 monclient: _renew_subs 2024-03-29T11:25:05.999 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:05.984+0000 7fcc05f3d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.84:6789/0 2024-03-29T11:25:05.999 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:05.984+0000 7fcc05f3d640 1 -- 172.21.15.84:0/2127700912 --> v1:172.21.15.84:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fcc00071c30 con 0x7fcc000b17d0 2024-03-29T11:25:05.999 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:05.985+0000 7fcc05f3d640 1 librados: init done 2024-03-29T11:25:06.000 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:05.985+0000 7fcbf7fff640 1 -- 172.21.15.84:0/2127700912 <== mon.0 v1:172.21.15.84:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fcbf0003c20 con 0x7fcc000b17d0 2024-03-29T11:25:06.000 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:05.985+0000 7fcbf7fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:25:06.000 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:05.985+0000 7fcbfd7fa640 4 set_mon_vals no callback set 2024-03-29T11:25:06.000 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:05.985+0000 7fcbf7fff640 1 -- 172.21.15.84:0/2127700912 <== mon.0 v1:172.21.15.84:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4236868694 0 0) 0x7fcbf0003f30 con 0x7fcc000b17d0 2024-03-29T11:25:06.000 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:05.985+0000 7fcbf7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:25:06.000 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:05.985+0000 7fcbf7fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:25:06.000 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:05.985+0000 7fcbf7fff640 10 monclient: dump: 2024-03-29T11:25:06.000 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T11:25:06.000 INFO:teuthology.misc.health.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:25:06.000 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T11:18:53.864025+0000 2024-03-29T11:25:06.000 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T11:18:53.864025+0000 2024-03-29T11:25:06.000 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T11:25:06.000 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:25:06.000 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T11:25:06.000 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T11:25:06.001 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T11:25:06.001 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:25:06.001 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:05.986+0000 7fcc05f3d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:25:06.001 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:05.986+0000 7fcc05f3d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:25:06.001 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:05.987+0000 7fcc05f3d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.84:6789/0 2024-03-29T11:25:06.001 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:05.987+0000 7fcc05f3d640 1 -- 172.21.15.84:0/2127700912 --> v1:172.21.15.84:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcbe4005080 con 0x7fcc000b17d0 2024-03-29T11:25:06.001 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:05.988+0000 7fcbf7fff640 1 -- 172.21.15.84:0/2127700912 <== mon.0 v1:172.21.15.84:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4033942909 0 0) 0x7fcbf0019f00 con 0x7fcc000b17d0 2024-03-29T11:25:06.001 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:05.989+0000 7fcbf7fff640 1 -- 172.21.15.84:0/2127700912 <== mon.0 v1:172.21.15.84:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2808139587 0 0) 0x7fcbf0083cb0 con 0x7fcc000b17d0 2024-03-29T11:25:06.001 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:05.989+0000 7fcbf7fff640 10 client.4443.objecter ms_dispatch 0x7fcc00000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:25:06.001 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:05.989+0000 7fcbf7fff640 3 client.4443.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:25:06.001 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:05.989+0000 7fcbf7fff640 3 client.4443.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:25:06.001 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:05.989+0000 7fcbf7fff640 20 client.4443.objecter dump_active .. 0 homeless 2024-03-29T11:25:06.001 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:05.989+0000 7fcbfeffd640 10 monclient: get_auth_request con 0x7fcbe006e700 auth_method 0 2024-03-29T11:25:06.001 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:05.990+0000 7fcbf7fff640 10 client.4443.objecter ms_handle_connect 0x7fcbe006e700 2024-03-29T11:25:06.002 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:05.996+0000 7fcbf7fff640 1 -- 172.21.15.84:0/2127700912 <== mon.0 v1:172.21.15.84:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fcbf004f590 con 0x7fcc000b17d0 2024-03-29T11:25:06.002 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:05.996+0000 7fcbf7fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:25:06.002 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:05.996+0000 7fcbf7fff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:25:06.328 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:06.345+0000 7fcc05f3d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:25:06.328 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:06.345+0000 7fcc05f3d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:25:06.329 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:06.345+0000 7fcc05f3d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.84:6789/0 2024-03-29T11:25:06.329 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:06.345+0000 7fcc05f3d640 1 -- 172.21.15.84:0/2127700912 --> v1:172.21.15.84:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fcbe4005330 con 0x7fcc000b17d0 2024-03-29T11:25:06.329 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:06.346+0000 7fcbf7fff640 1 -- 172.21.15.84:0/2127700912 <== mon.0 v1:172.21.15.84:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4209477549) 0x7fcbf00514c0 con 0x7fcc000b17d0 2024-03-29T11:25:06.329 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:06.346+0000 7fcbf7fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:25:06.329 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:06.346+0000 7fcbf7fff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:25:06.330 INFO:teuthology.misc.health.smithi084.stdout: 2024-03-29T11:25:06.330 INFO:teuthology.misc.health.smithi084.stdout:{"epoch":10,"fsid":"2c3e002f-9eb0-4860-853b-4ef1ef562213","created":"2024-03-29T11:19:32.029169+0000","modified":"2024-03-29T11:19:34.200299+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:34.200299+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"5c39e276-2034-4716-bbc0-57a634eefa92","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"cd580e99-109f-4bf1-88c0-a5987bc8b9ed","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"11e60a5a-051d-453f-a190-66387cb35592","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"2c329fbe-bbc4-4030-bf87-d711cad04679","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f4213d4a-bd36-4433-91ae-d7e2b02481d3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"f0cf44ea-6055-48af-805b-aed858102ff9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"d895eaa0-cc0b-4ee4-9b5a-2dfcd89a60b7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"9bc5a7bc-8f52-4db9-96d6-de5e68d52eaa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:25:06.334 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:06.349+0000 7fcc05f3d640 10 librados: watch_flush enter 2024-03-29T11:25:06.334 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:06.349+0000 7fcc05f3d640 10 librados: watch_flush exit 2024-03-29T11:25:06.334 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:06.349+0000 7fcc05f3d640 20 client.4443.objecter shutdown clearing up homeless session... 2024-03-29T11:25:06.334 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:06.349+0000 7fcc05f3d640 10 client.4443.objecter successfully canceled tick 2024-03-29T11:25:06.334 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:06.350+0000 7fcc05f3d640 1 -- 172.21.15.84:0/2127700912 >> v1:172.21.15.179:6816/2981134980 conn(0x7fcbe006e700 legacy=0x7fcbe0070bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:06.334 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:06.350+0000 7fcc05f3d640 10 monclient: shutdown 2024-03-29T11:25:06.334 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:06.350+0000 7fcc05f3d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:25:06.334 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:06.350+0000 7fcc05f3d640 1 -- 172.21.15.84:0/2127700912 >> v1:172.21.15.84:6789/0 conn(0x7fcc000b17d0 legacy=0x7fcc0006f5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:06.334 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:06.350+0000 7fcc05f3d640 1 -- 172.21.15.84:0/2127700912 shutdown_connections 2024-03-29T11:25:06.334 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:06.350+0000 7fcc05f3d640 1 -- 172.21.15.84:0/2127700912 >> 172.21.15.84:0/2127700912 conn(0x7fcc00066a00 msgr2=0x7fcc001409f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:25:06.334 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:06.350+0000 7fcc05f3d640 1 -- 172.21.15.84:0/2127700912 shutdown_connections 2024-03-29T11:25:06.335 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:06.350+0000 7fcc05f3d640 1 -- 172.21.15.84:0/2127700912 wait complete. 2024-03-29T11:25:06.335 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:06.351+0000 7fcc05f3d640 1 librados: shutdown 2024-03-29T11:25:06.348 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:25:13.155 DEBUG:teuthology.orchestra.run.smithi084:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:25:13.284 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:13.300+0000 7f1fca089640 10 monclient: get_monmap_and_config 2024-03-29T11:25:13.284 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:13.300+0000 7f1fca089640 10 monclient: build_initial_monmap 2024-03-29T11:25:13.284 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:13.300+0000 7f1fca089640 10 monclient: monmap: 2024-03-29T11:25:13.284 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-29T11:25:13.284 INFO:teuthology.misc.health.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:25:13.285 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T11:25:13.301549+0000 2024-03-29T11:25:13.285 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T11:25:13.301549+0000 2024-03-29T11:25:13.285 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-29T11:25:13.285 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:25:13.285 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-29T11:25:13.286 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.noname-c 2024-03-29T11:25:13.286 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-b 2024-03-29T11:25:13.286 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:25:13.307 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:13.301+0000 7f1fca089640 1 Processor -- start 2024-03-29T11:25:13.307 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:13.301+0000 7f1fca089640 1 -- start start 2024-03-29T11:25:13.307 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:13.301+0000 7f1fca089640 10 monclient: init 2024-03-29T11:25:13.307 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:13.302+0000 7f1fca089640 10 monclient: _reopen_session rank -1 2024-03-29T11:25:13.308 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:13.302+0000 7f1fca089640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:25:13.308 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:13.302+0000 7f1fca089640 10 monclient(hunting): picked mon.noname-b con 0x7f1fc41457f0 addr v1:172.21.15.84:6790/0 2024-03-29T11:25:13.308 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:13.302+0000 7f1fca089640 10 monclient(hunting): picked mon.noname-c con 0x7f1fc413f650 addr v1:172.21.15.179:6789/0 2024-03-29T11:25:13.308 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:13.302+0000 7f1fc37fe640 1 --1- >> v1:172.21.15.84:6790/0 conn(0x7f1fc41457f0 0x7f1fc413ef40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.84:6790/0 says I am v1:172.21.15.84:40100/0 (socket says 172.21.15.84:40100) 2024-03-29T11:25:13.308 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:13.302+0000 7f1fc37fe640 1 -- 172.21.15.84:0/1832800476 learned_addr learned my addr 172.21.15.84:0/1832800476 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-29T11:25:13.309 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:13.302+0000 7f1fca089640 10 monclient(hunting): picked mon.noname-a con 0x7f1fc40a9d70 addr v1:172.21.15.84:6789/0 2024-03-29T11:25:13.309 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:13.302+0000 7f1fca089640 1 -- 172.21.15.84:0/1832800476 --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1fc40aaef0 con 0x7f1fc40a9d70 2024-03-29T11:25:13.309 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:13.303+0000 7f1fca089640 1 -- 172.21.15.84:0/1832800476 --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1fc4145bd0 con 0x7f1fc413f650 2024-03-29T11:25:13.309 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:13.303+0000 7f1fca089640 1 -- 172.21.15.84:0/1832800476 --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1fc4145eb0 con 0x7f1fc41457f0 2024-03-29T11:25:13.310 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:13.303+0000 7f1fca089640 10 monclient(hunting): _renew_subs 2024-03-29T11:25:13.310 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:13.303+0000 7f1fca089640 10 monclient(hunting): authenticate will time out at 1174.597328s 2024-03-29T11:25:13.310 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:13.304+0000 7f1fc27fc640 1 -- 172.21.15.84:0/1832800476 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3741176170 0 0) 0x7f1fc4145eb0 con 0x7f1fc41457f0 2024-03-29T11:25:13.310 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:13.304+0000 7f1fc27fc640 10 monclient(hunting): _negotiate 2024-03-29T11:25:13.310 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:13.304+0000 7f1fc27fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:25:13.310 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:13.304+0000 7f1fc27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:25:13.310 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:13.304+0000 7f1fc27fc640 10 monclient(hunting): my global_id is 4445 2024-03-29T11:25:13.310 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:13.304+0000 7f1fc27fc640 1 -- 172.21.15.84:0/1832800476 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1fac003520 con 0x7f1fc41457f0 2024-03-29T11:25:13.310 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:13.304+0000 7f1fc27fc640 1 -- 172.21.15.84:0/1832800476 <== mon.2 v1:172.21.15.84:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1499292907 0 0) 0x7f1fac003520 con 0x7f1fc41457f0 2024-03-29T11:25:13.310 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:13.304+0000 7f1fc27fc640 1 -- 172.21.15.84:0/1832800476 >> v1:172.21.15.179:6789/0 conn(0x7f1fc413f650 legacy=0x7f1fc40b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:13.311 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:13.304+0000 7f1fc27fc640 1 -- 172.21.15.84:0/1832800476 >> v1:172.21.15.84:6789/0 conn(0x7f1fc40a9d70 legacy=0x7f1fc414d8c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:25:13.311 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:13.306+0000 7f1fc27fc640 10 monclient: _finish_hunting 0 2024-03-29T11:25:13.311 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:13.306+0000 7f1fc27fc640 1 monclient: found mon.noname-b 2024-03-29T11:25:13.311 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:13.306+0000 7f1fc27fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:25:13.311 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:13.306+0000 7f1fc27fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.84:6790/0 2024-03-29T11:25:13.311 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:13.306+0000 7f1fc27fc640 1 -- 172.21.15.84:0/1832800476 --> v1:172.21.15.84:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1fc4146340 con 0x7f1fc41457f0 2024-03-29T11:25:13.311 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:13.306+0000 7f1fc27fc640 10 monclient: _finish_auth 0 2024-03-29T11:25:13.311 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:13.306+0000 7f1fc27fc640 10 monclient: _check_auth_tickets 2024-03-29T11:25:13.311 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:13.306+0000 7f1fc27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:25:13.311 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:13.306+0000 7f1fc27fc640 1 -- 172.21.15.84:0/1832800476 <== mon.2 v1:172.21.15.84:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4236868694 0 0) 0x7f1fb4001690 con 0x7f1fc41457f0 2024-03-29T11:25:13.311 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:13.306+0000 7f1fc27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:25:13.311 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:13.306+0000 7f1fc27fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:25:13.311 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:13.306+0000 7f1fc27fc640 10 monclient: dump: 2024-03-29T11:25:13.311 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T11:25:13.311 INFO:teuthology.misc.health.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:25:13.312 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T11:18:53.864025+0000 2024-03-29T11:25:13.312 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T11:18:53.864025+0000 2024-03-29T11:25:13.312 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T11:25:13.312 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:25:13.312 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T11:25:13.312 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T11:25:13.312 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T11:25:13.312 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:25:13.312 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:13.306+0000 7f1fc27fc640 1 -- 172.21.15.84:0/1832800476 <== mon.2 v1:172.21.15.84:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1fb4001860 con 0x7f1fc41457f0 2024-03-29T11:25:13.312 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:13.306+0000 7f1fca089640 5 monclient: authenticate success, global_id 4445 2024-03-29T11:25:13.312 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:13.307+0000 7f1fca089640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:25:13.312 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:13.307+0000 7f1fc27fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:25:13.312 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:13.307+0000 7f1fca089640 10 monclient: get_monmap_and_config success 2024-03-29T11:25:13.312 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:13.307+0000 7f1fca089640 4 set_mon_vals no callback set 2024-03-29T11:25:13.312 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:13.307+0000 7f1fc27fc640 1 -- 172.21.15.84:0/1832800476 <== mon.2 v1:172.21.15.84:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4236868694 0 0) 0x7f1fb4001cb0 con 0x7f1fc41457f0 2024-03-29T11:25:13.313 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:13.307+0000 7f1fc27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:25:13.313 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:13.307+0000 7f1fc27fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:25:13.313 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:13.307+0000 7f1fc27fc640 10 monclient: dump: 2024-03-29T11:25:13.313 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T11:25:13.313 INFO:teuthology.misc.health.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:25:13.313 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T11:18:53.864025+0000 2024-03-29T11:25:13.313 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T11:18:53.864025+0000 2024-03-29T11:25:13.313 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T11:25:13.313 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:25:13.313 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T11:25:13.313 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T11:25:13.313 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T11:25:13.313 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:25:13.313 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:13.308+0000 7f1fca089640 10 monclient: shutdown 2024-03-29T11:25:13.313 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:13.308+0000 7f1fca089640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:25:13.314 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:13.308+0000 7f1fca089640 1 -- 172.21.15.84:0/1832800476 >> v1:172.21.15.84:6790/0 conn(0x7f1fc41457f0 legacy=0x7f1fc413ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:13.314 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:13.308+0000 7f1fca089640 1 -- 172.21.15.84:0/1832800476 shutdown_connections 2024-03-29T11:25:13.314 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:13.308+0000 7f1fca089640 1 -- 172.21.15.84:0/1832800476 >> 172.21.15.84:0/1832800476 conn(0x7f1fc4066a00 msgr2=0x7f1fc4066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:25:13.314 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:13.308+0000 7f1fca089640 1 -- 172.21.15.84:0/1832800476 shutdown_connections 2024-03-29T11:25:13.314 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:13.308+0000 7f1fca089640 1 -- 172.21.15.84:0/1832800476 wait complete. 2024-03-29T11:25:13.314 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:13.309+0000 7f1fca089640 10 monclient: build_initial_monmap 2024-03-29T11:25:13.314 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:13.309+0000 7f1fca089640 10 monclient: monmap: 2024-03-29T11:25:13.314 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-29T11:25:13.314 INFO:teuthology.misc.health.smithi084.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:25:13.314 INFO:teuthology.misc.health.smithi084.stderr:last_changed 0.000000 2024-03-29T11:25:13.314 INFO:teuthology.misc.health.smithi084.stderr:created 0.000000 2024-03-29T11:25:13.314 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-29T11:25:13.315 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:25:13.315 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-29T11:25:13.315 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.noname-b 2024-03-29T11:25:13.315 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-c 2024-03-29T11:25:13.315 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:25:13.315 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:13.309+0000 7f1fca089640 1 librados: starting msgr at 2024-03-29T11:25:13.315 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:13.309+0000 7f1fca089640 1 librados: starting objecter 2024-03-29T11:25:13.315 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:13.310+0000 7f1fca089640 1 Processor -- start 2024-03-29T11:25:13.315 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:13.310+0000 7f1fca089640 1 -- start start 2024-03-29T11:25:13.315 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:13.310+0000 7f1fca089640 1 librados: setting wanted keys 2024-03-29T11:25:13.315 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:13.310+0000 7f1fca089640 1 librados: calling monclient init 2024-03-29T11:25:13.315 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:13.310+0000 7f1fca089640 10 monclient: init 2024-03-29T11:25:13.315 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:13.311+0000 7f1fca089640 10 monclient: _reopen_session rank -1 2024-03-29T11:25:13.315 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:13.312+0000 7f1fca089640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:25:13.316 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:13.312+0000 7f1fca089640 10 monclient(hunting): picked mon.noname-a con 0x7f1fc40a9d70 addr v1:172.21.15.84:6789/0 2024-03-29T11:25:13.316 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:13.312+0000 7f1fca089640 10 monclient(hunting): picked mon.noname-b con 0x7f1fc413f650 addr v1:172.21.15.179:6789/0 2024-03-29T11:25:13.316 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:13.312+0000 7f1fca089640 10 monclient(hunting): picked mon.noname-c con 0x7f1fc41457f0 addr v1:172.21.15.84:6790/0 2024-03-29T11:25:13.316 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:13.312+0000 7f1fca089640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1fc41df000 con 0x7f1fc40a9d70 2024-03-29T11:25:13.316 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:13.312+0000 7f1fc3fff640 1 --1- >> v1:172.21.15.84:6790/0 conn(0x7f1fc41457f0 0x7f1fc41e6020 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.84:6790/0 says I am v1:172.21.15.84:41494/0 (socket says 172.21.15.84:41494) 2024-03-29T11:25:13.316 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:13.312+0000 7f1fc3fff640 1 -- 172.21.15.84:0/3454690786 learned_addr learned my addr 172.21.15.84:0/3454690786 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-29T11:25:13.316 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:13.312+0000 7f1fca089640 1 -- 172.21.15.84:0/3454690786 --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1fc40ab600 con 0x7f1fc413f650 2024-03-29T11:25:13.316 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:13.312+0000 7f1fca089640 1 -- 172.21.15.84:0/3454690786 --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1fc40ab8e0 con 0x7f1fc41457f0 2024-03-29T11:25:13.316 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:13.312+0000 7f1fca089640 10 monclient(hunting): _renew_subs 2024-03-29T11:25:13.316 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:13.312+0000 7f1fabfff640 10 client.?.objecter ms_handle_connect 0x7f1fc41457f0 2024-03-29T11:25:13.316 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:13.312+0000 7f1fabfff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:25:13.316 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:13.312+0000 7f1fca089640 10 monclient(hunting): authenticate will time out at 1174.606663s 2024-03-29T11:25:13.316 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:13.312+0000 7f1fabfff640 10 client.?.objecter ms_handle_connect 0x7f1fc413f650 2024-03-29T11:25:13.316 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:13.312+0000 7f1fabfff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:25:13.317 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:13.312+0000 7f1fabfff640 1 -- 172.21.15.84:0/3454690786 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3775914361 0 0) 0x7f1fc40ab8e0 con 0x7f1fc41457f0 2024-03-29T11:25:13.317 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:13.312+0000 7f1fabfff640 10 monclient(hunting): _negotiate 2024-03-29T11:25:13.317 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:13.312+0000 7f1fabfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:25:13.317 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:13.312+0000 7f1fabfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:25:13.317 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:13.312+0000 7f1fabfff640 10 monclient(hunting): my global_id is 4448 2024-03-29T11:25:13.317 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:13.312+0000 7f1fabfff640 1 -- 172.21.15.84:0/3454690786 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1fb0004eb0 con 0x7f1fc41457f0 2024-03-29T11:25:13.317 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:13.312+0000 7f1fabfff640 1 -- 172.21.15.84:0/3454690786 <== mon.1 v1:172.21.15.179:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2621923277 0 0) 0x7f1fc40ab600 con 0x7f1fc413f650 2024-03-29T11:25:13.317 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:13.312+0000 7f1fabfff640 10 monclient(hunting): _negotiate 2024-03-29T11:25:13.317 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:13.312+0000 7f1fabfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:25:13.317 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:13.312+0000 7f1fabfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:25:13.317 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:13.312+0000 7f1fabfff640 10 monclient(hunting): my global_id is 4405 2024-03-29T11:25:13.317 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:13.312+0000 7f1fabfff640 1 -- 172.21.15.84:0/3454690786 --> v1:172.21.15.179:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1fc40ab8e0 con 0x7f1fc413f650 2024-03-29T11:25:13.317 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:13.313+0000 7f1fabfff640 1 -- 172.21.15.84:0/3454690786 <== mon.2 v1:172.21.15.84:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2953152197 0 0) 0x7f1fb0004eb0 con 0x7f1fc41457f0 2024-03-29T11:25:13.317 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:13.313+0000 7f1fabfff640 1 -- 172.21.15.84:0/3454690786 --> v1:172.21.15.84:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1fc40ab600 con 0x7f1fc41457f0 2024-03-29T11:25:13.317 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:13.313+0000 7f1fabfff640 1 -- 172.21.15.84:0/3454690786 <== mon.2 v1:172.21.15.84:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4236868694 0 0) 0x7f1fbc001c60 con 0x7f1fc41457f0 2024-03-29T11:25:13.318 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:13.313+0000 7f1fabfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:25:13.318 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:13.313+0000 7f1fabfff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:25:13.318 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:13.313+0000 7f1fabfff640 10 monclient(hunting): dump: 2024-03-29T11:25:13.318 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T11:25:13.318 INFO:teuthology.misc.health.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:25:13.318 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T11:18:53.864025+0000 2024-03-29T11:25:13.318 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T11:18:53.864025+0000 2024-03-29T11:25:13.318 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T11:25:13.318 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:25:13.318 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T11:25:13.318 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T11:25:13.318 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T11:25:13.318 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:25:13.318 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:13.313+0000 7f1fabfff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:25:13.319 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:13.313+0000 7f1fabfff640 1 -- 172.21.15.84:0/3454690786 <== mon.2 v1:172.21.15.84:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 621751630 0 0) 0x7f1fc40ab600 con 0x7f1fc41457f0 2024-03-29T11:25:13.319 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:13.313+0000 7f1fabfff640 1 -- 172.21.15.84:0/3454690786 >> v1:172.21.15.179:6789/0 conn(0x7f1fc413f650 legacy=0x7f1fc406fed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:13.319 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:13.313+0000 7f1fabfff640 1 -- 172.21.15.84:0/3454690786 >> v1:172.21.15.84:6789/0 conn(0x7f1fc40a9d70 legacy=0x7f1fc41df490 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:25:13.319 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:13.314+0000 7f1fabfff640 10 monclient: _finish_hunting 0 2024-03-29T11:25:13.319 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:13.314+0000 7f1fabfff640 1 monclient: found mon.c 2024-03-29T11:25:13.319 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:13.314+0000 7f1fabfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:25:13.319 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:13.314+0000 7f1fabfff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-29T11:25:13.319 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:13.314+0000 7f1fabfff640 1 -- 172.21.15.84:0/3454690786 --> v1:172.21.15.84:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1fc413b1d0 con 0x7f1fc41457f0 2024-03-29T11:25:13.319 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:13.314+0000 7f1fabfff640 10 monclient: _finish_auth 0 2024-03-29T11:25:13.319 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:13.314+0000 7f1fabfff640 10 monclient: _check_auth_tickets 2024-03-29T11:25:13.319 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:13.314+0000 7f1fabfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:25:13.319 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:13.314+0000 7f1fca089640 5 monclient: authenticate success, global_id 4448 2024-03-29T11:25:13.319 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:13.314+0000 7f1fca089640 10 monclient: _renew_subs 2024-03-29T11:25:13.319 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:13.314+0000 7f1fabfff640 1 -- 172.21.15.84:0/3454690786 <== mon.2 v1:172.21.15.84:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1fbc003970 con 0x7f1fc41457f0 2024-03-29T11:25:13.319 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:13.314+0000 7f1fca089640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-29T11:25:13.320 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:13.315+0000 7f1fca089640 1 -- 172.21.15.84:0/3454690786 --> v1:172.21.15.84:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1fc41deae0 con 0x7f1fc41457f0 2024-03-29T11:25:13.320 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:13.315+0000 7f1fca089640 10 client.4448.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:25:13.320 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:13.315+0000 7f1fca089640 10 monclient: _renew_subs 2024-03-29T11:25:13.320 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:13.315+0000 7f1fca089640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-29T11:25:13.320 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:13.315+0000 7f1fca089640 1 -- 172.21.15.84:0/3454690786 --> v1:172.21.15.84:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f1fc4142b30 con 0x7f1fc41457f0 2024-03-29T11:25:13.320 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:13.315+0000 7f1fca089640 1 librados: init done 2024-03-29T11:25:13.320 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:13.315+0000 7f1fabfff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:25:13.320 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:13.315+0000 7f1fabfff640 1 -- 172.21.15.84:0/3454690786 <== mon.2 v1:172.21.15.84:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4236868694 0 0) 0x7f1fbc003d40 con 0x7f1fc41457f0 2024-03-29T11:25:13.320 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:13.315+0000 7f1fabfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:25:13.320 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:13.315+0000 7f1fabfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:25:13.320 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:13.315+0000 7f1fc1ffb640 4 set_mon_vals no callback set 2024-03-29T11:25:13.320 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:13.315+0000 7f1fabfff640 10 monclient: dump: 2024-03-29T11:25:13.321 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T11:25:13.321 INFO:teuthology.misc.health.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:25:13.321 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T11:18:53.864025+0000 2024-03-29T11:25:13.321 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T11:18:53.864025+0000 2024-03-29T11:25:13.321 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T11:25:13.321 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:25:13.321 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T11:25:13.321 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T11:25:13.321 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T11:25:13.321 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:25:13.322 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:13.317+0000 7f1fabfff640 1 -- 172.21.15.84:0/3454690786 <== mon.2 v1:172.21.15.84:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4033942909 0 0) 0x7f1fbc003ea0 con 0x7f1fc41457f0 2024-03-29T11:25:13.322 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:13.318+0000 7f1fabfff640 1 -- 172.21.15.84:0/3454690786 <== mon.2 v1:172.21.15.84:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2808139587 0 0) 0x7f1fbc084db0 con 0x7f1fc41457f0 2024-03-29T11:25:13.322 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:13.318+0000 7f1fabfff640 10 client.4448.objecter ms_dispatch 0x7f1fc4000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:25:13.322 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:13.318+0000 7f1fc37fe640 10 monclient: get_auth_request con 0x7f1fb006dcb0 auth_method 0 2024-03-29T11:25:13.322 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:13.318+0000 7f1fabfff640 3 client.4448.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:25:13.322 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:13.319+0000 7f1fabfff640 3 client.4448.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:25:13.322 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:13.319+0000 7f1fabfff640 20 client.4448.objecter dump_active .. 0 homeless 2024-03-29T11:25:13.322 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:13.319+0000 7f1fabfff640 10 client.4448.objecter ms_handle_connect 0x7f1fb006dcb0 2024-03-29T11:25:13.322 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:13.319+0000 7f1fca089640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:25:13.322 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:13.319+0000 7f1fca089640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:25:13.322 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:13.319+0000 7f1fca089640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-29T11:25:13.322 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:13.319+0000 7f1fca089640 1 -- 172.21.15.84:0/3454690786 --> v1:172.21.15.84:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1fa4005080 con 0x7f1fc41457f0 2024-03-29T11:25:13.323 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:13.327+0000 7f1fabfff640 1 -- 172.21.15.84:0/3454690786 <== mon.2 v1:172.21.15.84:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f1fbc050690 con 0x7f1fc41457f0 2024-03-29T11:25:13.323 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:13.328+0000 7f1fabfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:25:13.323 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:13.328+0000 7f1fabfff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:25:13.615 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:13.631+0000 7f1fca089640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:25:13.615 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:13.631+0000 7f1fca089640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:25:13.615 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:13.631+0000 7f1fca089640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-29T11:25:13.615 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:13.631+0000 7f1fca089640 1 -- 172.21.15.84:0/3454690786 --> v1:172.21.15.84:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1fa4005330 con 0x7f1fc41457f0 2024-03-29T11:25:13.615 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:13.632+0000 7f1fabfff640 1 -- 172.21.15.84:0/3454690786 <== mon.2 v1:172.21.15.84:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4209477549) 0x7f1fbc0525c0 con 0x7f1fc41457f0 2024-03-29T11:25:13.615 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:13.632+0000 7f1fabfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:25:13.615 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:13.632+0000 7f1fabfff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:25:13.616 INFO:teuthology.misc.health.smithi084.stdout: 2024-03-29T11:25:13.616 INFO:teuthology.misc.health.smithi084.stdout:{"epoch":10,"fsid":"2c3e002f-9eb0-4860-853b-4ef1ef562213","created":"2024-03-29T11:19:32.029169+0000","modified":"2024-03-29T11:19:34.200299+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:34.200299+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"5c39e276-2034-4716-bbc0-57a634eefa92","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"cd580e99-109f-4bf1-88c0-a5987bc8b9ed","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"11e60a5a-051d-453f-a190-66387cb35592","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"2c329fbe-bbc4-4030-bf87-d711cad04679","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f4213d4a-bd36-4433-91ae-d7e2b02481d3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"f0cf44ea-6055-48af-805b-aed858102ff9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"d895eaa0-cc0b-4ee4-9b5a-2dfcd89a60b7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"9bc5a7bc-8f52-4db9-96d6-de5e68d52eaa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:25:13.620 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:13.635+0000 7f1fca089640 10 librados: watch_flush enter 2024-03-29T11:25:13.620 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:13.635+0000 7f1fca089640 10 librados: watch_flush exit 2024-03-29T11:25:13.620 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:13.635+0000 7f1fca089640 20 client.4448.objecter shutdown clearing up homeless session... 2024-03-29T11:25:13.620 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:13.635+0000 7f1fca089640 10 client.4448.objecter successfully canceled tick 2024-03-29T11:25:13.620 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:13.635+0000 7f1fca089640 1 -- 172.21.15.84:0/3454690786 >> v1:172.21.15.179:6816/2981134980 conn(0x7f1fb006dcb0 legacy=0x7f1fb0070170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:13.620 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:13.635+0000 7f1fca089640 10 monclient: shutdown 2024-03-29T11:25:13.620 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:13.635+0000 7f1fca089640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:25:13.621 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:13.635+0000 7f1fca089640 1 -- 172.21.15.84:0/3454690786 >> v1:172.21.15.84:6790/0 conn(0x7f1fc41457f0 legacy=0x7f1fc41e6020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:13.621 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:13.635+0000 7f1fca089640 1 -- 172.21.15.84:0/3454690786 shutdown_connections 2024-03-29T11:25:13.621 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:13.635+0000 7f1fca089640 1 -- 172.21.15.84:0/3454690786 >> 172.21.15.84:0/3454690786 conn(0x7f1fc4066a00 msgr2=0x7f1fc40b5560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:25:13.621 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:13.636+0000 7f1fca089640 1 -- 172.21.15.84:0/3454690786 shutdown_connections 2024-03-29T11:25:13.621 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:13.636+0000 7f1fca089640 1 -- 172.21.15.84:0/3454690786 wait complete. 2024-03-29T11:25:13.621 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:13.636+0000 7f1fca089640 1 librados: shutdown 2024-03-29T11:25:13.633 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:25:20.440 DEBUG:teuthology.orchestra.run.smithi084:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:25:20.585 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:20.601+0000 7f18e6cd9640 10 monclient: get_monmap_and_config 2024-03-29T11:25:20.585 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:20.601+0000 7f18e6cd9640 10 monclient: build_initial_monmap 2024-03-29T11:25:20.585 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:20.601+0000 7f18e6cd9640 10 monclient: monmap: 2024-03-29T11:25:20.585 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-29T11:25:20.585 INFO:teuthology.misc.health.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:25:20.586 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T11:25:20.602426+0000 2024-03-29T11:25:20.586 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T11:25:20.602426+0000 2024-03-29T11:25:20.586 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-29T11:25:20.586 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:25:20.586 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-29T11:25:20.586 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.noname-c 2024-03-29T11:25:20.586 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-b 2024-03-29T11:25:20.586 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:25:20.587 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:20.602+0000 7f18e6cd9640 1 Processor -- start 2024-03-29T11:25:20.587 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:20.603+0000 7f18e6cd9640 1 -- start start 2024-03-29T11:25:20.587 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:20.603+0000 7f18e6cd9640 10 monclient: init 2024-03-29T11:25:20.587 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:20.604+0000 7f18e6cd9640 10 monclient: _reopen_session rank -1 2024-03-29T11:25:20.587 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:20.604+0000 7f18e6cd9640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:25:20.588 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:20.604+0000 7f18e6cd9640 10 monclient(hunting): picked mon.noname-b con 0x7f18d80389e0 addr v1:172.21.15.84:6790/0 2024-03-29T11:25:20.588 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:20.604+0000 7f18e6cd9640 10 monclient(hunting): picked mon.noname-a con 0x7f18d8038dc0 addr v1:172.21.15.84:6789/0 2024-03-29T11:25:20.588 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:20.604+0000 7f18e6cd9640 10 monclient(hunting): picked mon.noname-c con 0x7f18d8030660 addr v1:172.21.15.179:6789/0 2024-03-29T11:25:20.588 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:20.604+0000 7f18e6cd9640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f18d80c54c0 con 0x7f18d8038dc0 2024-03-29T11:25:20.588 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:20.604+0000 7f18e6cd9640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f18d8036430 con 0x7f18d8030660 2024-03-29T11:25:20.588 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:20.604+0000 7f18e6cd9640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f18d8039840 con 0x7f18d80389e0 2024-03-29T11:25:20.588 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:20.604+0000 7f18e6cd9640 10 monclient(hunting): _renew_subs 2024-03-29T11:25:20.588 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:20.604+0000 7f18e6cd9640 10 monclient(hunting): authenticate will time out at 1181.898274s 2024-03-29T11:25:20.588 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:20.604+0000 7f18e5cd7640 1 --1- >> v1:172.21.15.84:6790/0 conn(0x7f18d80389e0 0x7f18d8033520 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.84:6790/0 says I am v1:172.21.15.84:59486/0 (socket says 172.21.15.84:59486) 2024-03-29T11:25:20.589 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:20.604+0000 7f18e5cd7640 1 -- 172.21.15.84:0/2738401845 learned_addr learned my addr 172.21.15.84:0/2738401845 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-29T11:25:20.590 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:20.606+0000 7f18e4cd5640 1 -- 172.21.15.84:0/2738401845 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1610855295 0 0) 0x7f18d8039840 con 0x7f18d80389e0 2024-03-29T11:25:20.590 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:20.606+0000 7f18e4cd5640 10 monclient(hunting): _negotiate 2024-03-29T11:25:20.590 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:20.606+0000 7f18e4cd5640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:25:20.590 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:20.606+0000 7f18e4cd5640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:25:20.590 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:20.606+0000 7f18e4cd5640 10 monclient(hunting): my global_id is 4451 2024-03-29T11:25:20.590 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:20.606+0000 7f18e4cd5640 1 -- 172.21.15.84:0/2738401845 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f18cc003520 con 0x7f18d80389e0 2024-03-29T11:25:20.591 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:20.607+0000 7f18e4cd5640 1 -- 172.21.15.84:0/2738401845 <== mon.2 v1:172.21.15.84:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 936159311 0 0) 0x7f18cc003520 con 0x7f18d80389e0 2024-03-29T11:25:20.591 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:20.608+0000 7f18e4cd5640 1 -- 172.21.15.84:0/2738401845 >> v1:172.21.15.179:6789/0 conn(0x7f18d8030660 legacy=0x7f18d80d3170 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:25:20.596 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:20.609+0000 7f18e4cd5640 1 -- 172.21.15.84:0/2738401845 >> v1:172.21.15.84:6789/0 conn(0x7f18d8038dc0 legacy=0x7f18d80cadd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:20.596 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:20.609+0000 7f18e4cd5640 10 monclient: _finish_hunting 0 2024-03-29T11:25:20.596 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:20.609+0000 7f18e4cd5640 1 monclient: found mon.noname-b 2024-03-29T11:25:20.596 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:20.609+0000 7f18e4cd5640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:25:20.596 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:20.609+0000 7f18e4cd5640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.84:6790/0 2024-03-29T11:25:20.596 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:20.609+0000 7f18e4cd5640 1 -- 172.21.15.84:0/2738401845 --> v1:172.21.15.84:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18d8035690 con 0x7f18d80389e0 2024-03-29T11:25:20.596 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:20.609+0000 7f18e4cd5640 10 monclient: _finish_auth 0 2024-03-29T11:25:20.596 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:20.609+0000 7f18e4cd5640 10 monclient: _check_auth_tickets 2024-03-29T11:25:20.596 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:20.609+0000 7f18e4cd5640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:25:20.596 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:20.610+0000 7f18e6cd9640 5 monclient: authenticate success, global_id 4451 2024-03-29T11:25:20.596 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:20.610+0000 7f18e6cd9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:25:20.596 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:20.610+0000 7f18e4cd5640 1 -- 172.21.15.84:0/2738401845 <== mon.2 v1:172.21.15.84:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4236868694 0 0) 0x7f18d0001690 con 0x7f18d80389e0 2024-03-29T11:25:20.596 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:20.610+0000 7f18e4cd5640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:25:20.597 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:20.610+0000 7f18e4cd5640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:25:20.597 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:20.610+0000 7f18e4cd5640 10 monclient: dump: 2024-03-29T11:25:20.597 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T11:25:20.597 INFO:teuthology.misc.health.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:25:20.597 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T11:18:53.864025+0000 2024-03-29T11:25:20.597 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T11:18:53.864025+0000 2024-03-29T11:25:20.597 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T11:25:20.597 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:25:20.597 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T11:25:20.597 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T11:25:20.597 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T11:25:20.597 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:25:20.597 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:20.610+0000 7f18e6cd9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:25:20.597 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:20.611+0000 7f18e4cd5640 1 -- 172.21.15.84:0/2738401845 <== mon.2 v1:172.21.15.84:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f18d0001860 con 0x7f18d80389e0 2024-03-29T11:25:20.597 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:20.611+0000 7f18e4cd5640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:25:20.598 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:20.611+0000 7f18e4cd5640 1 -- 172.21.15.84:0/2738401845 <== mon.2 v1:172.21.15.84:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4236868694 0 0) 0x7f18d0001c70 con 0x7f18d80389e0 2024-03-29T11:25:20.598 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:20.611+0000 7f18e4cd5640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:25:20.598 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:20.611+0000 7f18e4cd5640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:25:20.598 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:20.611+0000 7f18e4cd5640 10 monclient: dump: 2024-03-29T11:25:20.598 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T11:25:20.598 INFO:teuthology.misc.health.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:25:20.598 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T11:18:53.864025+0000 2024-03-29T11:25:20.598 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T11:18:53.864025+0000 2024-03-29T11:25:20.598 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T11:25:20.598 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:25:20.598 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T11:25:20.598 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T11:25:20.598 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T11:25:20.599 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:25:20.599 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:20.611+0000 7f18e6cd9640 10 monclient: get_monmap_and_config success 2024-03-29T11:25:20.599 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:20.611+0000 7f18e6cd9640 4 set_mon_vals no callback set 2024-03-29T11:25:20.599 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:20.612+0000 7f18e6cd9640 10 monclient: shutdown 2024-03-29T11:25:20.599 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:20.612+0000 7f18e6cd9640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:25:20.599 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:20.612+0000 7f18e6cd9640 1 -- 172.21.15.84:0/2738401845 >> v1:172.21.15.84:6790/0 conn(0x7f18d80389e0 legacy=0x7f18d8033520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:20.600 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:20.614+0000 7f18e6cd9640 1 -- 172.21.15.84:0/2738401845 shutdown_connections 2024-03-29T11:25:20.600 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:20.614+0000 7f18e6cd9640 1 -- 172.21.15.84:0/2738401845 >> 172.21.15.84:0/2738401845 conn(0x7f18d8005c00 msgr2=0x7f18d8006010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:25:20.600 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:20.615+0000 7f18e6cd9640 1 -- 172.21.15.84:0/2738401845 shutdown_connections 2024-03-29T11:25:20.600 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:20.615+0000 7f18e6cd9640 1 -- 172.21.15.84:0/2738401845 wait complete. 2024-03-29T11:25:20.603 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:20.616+0000 7f18e6cd9640 10 monclient: build_initial_monmap 2024-03-29T11:25:20.603 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:20.616+0000 7f18e6cd9640 10 monclient: monmap: 2024-03-29T11:25:20.603 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-29T11:25:20.603 INFO:teuthology.misc.health.smithi084.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:25:20.603 INFO:teuthology.misc.health.smithi084.stderr:last_changed 0.000000 2024-03-29T11:25:20.603 INFO:teuthology.misc.health.smithi084.stderr:created 0.000000 2024-03-29T11:25:20.604 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-29T11:25:20.604 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:25:20.604 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-29T11:25:20.604 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.noname-b 2024-03-29T11:25:20.604 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-c 2024-03-29T11:25:20.604 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:25:20.604 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:20.616+0000 7f18e6cd9640 1 librados: starting msgr at 2024-03-29T11:25:20.604 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:20.616+0000 7f18e6cd9640 1 librados: starting objecter 2024-03-29T11:25:20.604 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:20.616+0000 7f18e6cd9640 1 Processor -- start 2024-03-29T11:25:20.604 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:20.616+0000 7f18e6cd9640 1 -- start start 2024-03-29T11:25:20.604 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:20.616+0000 7f18e6cd9640 1 librados: setting wanted keys 2024-03-29T11:25:20.604 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:20.616+0000 7f18e6cd9640 1 librados: calling monclient init 2024-03-29T11:25:20.604 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:20.616+0000 7f18e6cd9640 10 monclient: init 2024-03-29T11:25:20.604 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:20.617+0000 7f18e6cd9640 10 monclient: _reopen_session rank -1 2024-03-29T11:25:20.605 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:20.617+0000 7f18e6cd9640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:25:20.605 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:20.619+0000 7f18e6cd9640 10 monclient(hunting): picked mon.noname-b con 0x7f18d8030660 addr v1:172.21.15.179:6789/0 2024-03-29T11:25:20.605 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:20.619+0000 7f18e6cd9640 10 monclient(hunting): picked mon.noname-a con 0x7f18d80389e0 addr v1:172.21.15.84:6789/0 2024-03-29T11:25:20.605 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:20.619+0000 7f18e6cd9640 10 monclient(hunting): picked mon.noname-c con 0x7f18d80c7a70 addr v1:172.21.15.84:6790/0 2024-03-29T11:25:20.605 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:20.619+0000 7f18e6cd9640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f18d800a220 con 0x7f18d80389e0 2024-03-29T11:25:20.605 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:20.619+0000 7f18e6cd9640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f18d802fc50 con 0x7f18d8030660 2024-03-29T11:25:20.605 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:20.619+0000 7f18e6cd9640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f18d802f8c0 con 0x7f18d80c7a70 2024-03-29T11:25:20.605 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:20.619+0000 7f18e6cd9640 10 monclient(hunting): _renew_subs 2024-03-29T11:25:20.605 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:20.619+0000 7f18e6cd9640 10 monclient(hunting): authenticate will time out at 1181.913579s 2024-03-29T11:25:20.614 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:20.620+0000 7f18e64d8640 1 --1- >> v1:172.21.15.84:6790/0 conn(0x7f18d80c7a70 0x7f18d802d250 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.84:6790/0 says I am v1:172.21.15.84:59962/0 (socket says 172.21.15.84:59962) 2024-03-29T11:25:20.614 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:20.620+0000 7f18e64d8640 1 -- 172.21.15.84:0/4036169308 learned_addr learned my addr 172.21.15.84:0/4036169308 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-29T11:25:20.614 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:20.621+0000 7f18d67fc640 10 client.?.objecter ms_handle_connect 0x7f18d80c7a70 2024-03-29T11:25:20.614 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:20.621+0000 7f18d67fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:25:20.614 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:20.621+0000 7f18d67fc640 1 -- 172.21.15.84:0/4036169308 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2191232863 0 0) 0x7f18d802f8c0 con 0x7f18d80c7a70 2024-03-29T11:25:20.614 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:20.621+0000 7f18d67fc640 10 monclient(hunting): _negotiate 2024-03-29T11:25:20.614 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:20.621+0000 7f18d67fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:25:20.614 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:20.621+0000 7f18d67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:25:20.615 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:20.621+0000 7f18d67fc640 10 monclient(hunting): my global_id is 4454 2024-03-29T11:25:20.615 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:20.621+0000 7f18d67fc640 1 -- 172.21.15.84:0/4036169308 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f18c4004eb0 con 0x7f18d80c7a70 2024-03-29T11:25:20.615 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:20.621+0000 7f18d67fc640 10 client.?.objecter ms_handle_connect 0x7f18d8030660 2024-03-29T11:25:20.615 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:20.621+0000 7f18d67fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:25:20.615 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:20.621+0000 7f18d67fc640 1 -- 172.21.15.84:0/4036169308 <== mon.1 v1:172.21.15.179:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2941302186 0 0) 0x7f18d802fc50 con 0x7f18d8030660 2024-03-29T11:25:20.615 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:20.622+0000 7f18d67fc640 10 monclient(hunting): _negotiate 2024-03-29T11:25:20.615 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:20.622+0000 7f18d67fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:25:20.615 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:20.622+0000 7f18d67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:25:20.615 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:20.622+0000 7f18d67fc640 10 monclient(hunting): my global_id is 4408 2024-03-29T11:25:20.615 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:20.622+0000 7f18d67fc640 1 -- 172.21.15.84:0/4036169308 --> v1:172.21.15.179:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f18d802f8c0 con 0x7f18d8030660 2024-03-29T11:25:20.615 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:20.622+0000 7f18d67fc640 1 -- 172.21.15.84:0/4036169308 <== mon.2 v1:172.21.15.84:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4170281512 0 0) 0x7f18c4004eb0 con 0x7f18d80c7a70 2024-03-29T11:25:20.615 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:20.622+0000 7f18d67fc640 1 -- 172.21.15.84:0/4036169308 --> v1:172.21.15.84:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f18d802fc50 con 0x7f18d80c7a70 2024-03-29T11:25:20.615 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:20.622+0000 7f18d67fc640 1 -- 172.21.15.84:0/4036169308 <== mon.2 v1:172.21.15.84:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4236868694 0 0) 0x7f18dc0017f0 con 0x7f18d80c7a70 2024-03-29T11:25:20.615 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:20.622+0000 7f18d67fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:25:20.616 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:20.622+0000 7f18d67fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:25:20.616 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:20.622+0000 7f18d67fc640 10 monclient(hunting): dump: 2024-03-29T11:25:20.616 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T11:25:20.616 INFO:teuthology.misc.health.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:25:20.616 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T11:18:53.864025+0000 2024-03-29T11:25:20.616 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T11:18:53.864025+0000 2024-03-29T11:25:20.616 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T11:25:20.616 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:25:20.616 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T11:25:20.616 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T11:25:20.616 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T11:25:20.616 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:25:20.616 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:20.622+0000 7f18d67fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:25:20.616 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:20.622+0000 7f18d67fc640 10 client.?.objecter ms_handle_connect 0x7f18d80389e0 2024-03-29T11:25:20.617 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:20.622+0000 7f18d67fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:25:20.617 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:20.622+0000 7f18d67fc640 1 -- 172.21.15.84:0/4036169308 <== mon.0 v1:172.21.15.84:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3392074541 0 0) 0x7f18d800a220 con 0x7f18d80389e0 2024-03-29T11:25:20.617 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:20.622+0000 7f18d67fc640 10 monclient(hunting): _negotiate 2024-03-29T11:25:20.617 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:20.622+0000 7f18d67fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:25:20.617 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:20.622+0000 7f18d67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:25:20.617 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:20.622+0000 7f18d67fc640 10 monclient(hunting): my global_id is 4449 2024-03-29T11:25:20.617 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:20.622+0000 7f18d67fc640 1 -- 172.21.15.84:0/4036169308 --> v1:172.21.15.84:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f18c4004eb0 con 0x7f18d80389e0 2024-03-29T11:25:20.617 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:20.622+0000 7f18d67fc640 1 -- 172.21.15.84:0/4036169308 <== mon.1 v1:172.21.15.179:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2100383762 0 0) 0x7f18d802f8c0 con 0x7f18d8030660 2024-03-29T11:25:20.617 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:20.623+0000 7f18d67fc640 1 -- 172.21.15.84:0/4036169308 --> v1:172.21.15.179:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f18d800a220 con 0x7f18d8030660 2024-03-29T11:25:20.617 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:20.623+0000 7f18d67fc640 1 -- 172.21.15.84:0/4036169308 <== mon.1 v1:172.21.15.179:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4236868694 0 0) 0x7f18d0001b70 con 0x7f18d8030660 2024-03-29T11:25:20.617 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:20.623+0000 7f18d67fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:25:20.617 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:20.623+0000 7f18d67fc640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:25:20.617 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:20.623+0000 7f18d67fc640 10 monclient(hunting): dump: 2024-03-29T11:25:20.618 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T11:25:20.618 INFO:teuthology.misc.health.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:25:20.618 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T11:18:53.864025+0000 2024-03-29T11:25:20.618 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T11:18:53.864025+0000 2024-03-29T11:25:20.618 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T11:25:20.618 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:25:20.618 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T11:25:20.618 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T11:25:20.618 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T11:25:20.618 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:25:20.618 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:20.623+0000 7f18d67fc640 1 -- 172.21.15.84:0/4036169308 <== mon.0 v1:172.21.15.84:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3109614725 0 0) 0x7f18c4004eb0 con 0x7f18d80389e0 2024-03-29T11:25:20.618 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:20.623+0000 7f18d67fc640 1 -- 172.21.15.84:0/4036169308 --> v1:172.21.15.84:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f18d802f8c0 con 0x7f18d80389e0 2024-03-29T11:25:20.618 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:20.623+0000 7f18d67fc640 1 -- 172.21.15.84:0/4036169308 <== mon.2 v1:172.21.15.84:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3828049169 0 0) 0x7f18d802fc50 con 0x7f18d80c7a70 2024-03-29T11:25:20.618 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:20.623+0000 7f18d67fc640 1 -- 172.21.15.84:0/4036169308 >> v1:172.21.15.179:6789/0 conn(0x7f18d8030660 legacy=0x7f18d802f1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:20.618 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:20.623+0000 7f18d67fc640 1 -- 172.21.15.84:0/4036169308 >> v1:172.21.15.84:6789/0 conn(0x7f18d80389e0 legacy=0x7f18d8166680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:20.619 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:20.623+0000 7f18d67fc640 10 monclient: _finish_hunting 0 2024-03-29T11:25:20.619 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:20.623+0000 7f18d67fc640 1 monclient: found mon.c 2024-03-29T11:25:20.619 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:20.623+0000 7f18d67fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:25:20.619 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:20.623+0000 7f18d67fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-29T11:25:20.619 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:20.623+0000 7f18d67fc640 1 -- 172.21.15.84:0/4036169308 --> v1:172.21.15.84:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18d8166d90 con 0x7f18d80c7a70 2024-03-29T11:25:20.619 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:20.623+0000 7f18d67fc640 10 monclient: _finish_auth 0 2024-03-29T11:25:20.619 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:20.623+0000 7f18d67fc640 10 monclient: _check_auth_tickets 2024-03-29T11:25:20.619 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:20.623+0000 7f18d67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:25:20.619 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:20.623+0000 7f18e6cd9640 5 monclient: authenticate success, global_id 4454 2024-03-29T11:25:20.619 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:20.623+0000 7f18e6cd9640 10 monclient: _renew_subs 2024-03-29T11:25:20.619 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:20.623+0000 7f18e6cd9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-29T11:25:20.619 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:20.623+0000 7f18e6cd9640 1 -- 172.21.15.84:0/4036169308 --> v1:172.21.15.84:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f18d8167040 con 0x7f18d80c7a70 2024-03-29T11:25:20.619 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:20.623+0000 7f18e6cd9640 10 client.4454.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:25:20.619 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:20.623+0000 7f18e6cd9640 10 monclient: _renew_subs 2024-03-29T11:25:20.620 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:20.623+0000 7f18e6cd9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-29T11:25:20.620 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:20.623+0000 7f18e6cd9640 1 -- 172.21.15.84:0/4036169308 --> v1:172.21.15.84:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f18d802db40 con 0x7f18d80c7a70 2024-03-29T11:25:20.620 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:20.623+0000 7f18e6cd9640 1 librados: init done 2024-03-29T11:25:20.620 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:20.624+0000 7f18d67fc640 1 -- 172.21.15.84:0/4036169308 <== mon.2 v1:172.21.15.84:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f18dc001950 con 0x7f18d80c7a70 2024-03-29T11:25:20.620 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:20.624+0000 7f18d67fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:25:20.620 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:20.624+0000 7f18d7fff640 4 set_mon_vals no callback set 2024-03-29T11:25:20.620 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:20.625+0000 7f18e6cd9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:25:20.620 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:20.625+0000 7f18d67fc640 1 -- 172.21.15.84:0/4036169308 <== mon.2 v1:172.21.15.84:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4236868694 0 0) 0x7f18dc0048a0 con 0x7f18d80c7a70 2024-03-29T11:25:20.620 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:20.625+0000 7f18d67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:25:20.620 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:20.625+0000 7f18d67fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:25:20.620 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:20.625+0000 7f18d67fc640 10 monclient: dump: 2024-03-29T11:25:20.621 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T11:25:20.621 INFO:teuthology.misc.health.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:25:20.621 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T11:18:53.864025+0000 2024-03-29T11:25:20.621 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T11:18:53.864025+0000 2024-03-29T11:25:20.621 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T11:25:20.621 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:25:20.621 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T11:25:20.621 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T11:25:20.621 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T11:25:20.621 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:25:20.621 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:20.626+0000 7f18d67fc640 1 -- 172.21.15.84:0/4036169308 <== mon.2 v1:172.21.15.84:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4033942909 0 0) 0x7f18dc01a870 con 0x7f18d80c7a70 2024-03-29T11:25:20.621 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:20.627+0000 7f18e6cd9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:25:20.621 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:20.627+0000 7f18e6cd9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-29T11:25:20.621 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:20.627+0000 7f18e6cd9640 1 -- 172.21.15.84:0/4036169308 --> v1:172.21.15.84:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f18c8005080 con 0x7f18d80c7a70 2024-03-29T11:25:20.622 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:20.627+0000 7f18e5cd7640 10 monclient: get_auth_request con 0x7f18c406ded0 auth_method 0 2024-03-29T11:25:20.622 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:20.627+0000 7f18d67fc640 1 -- 172.21.15.84:0/4036169308 <== mon.2 v1:172.21.15.84:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2808139587 0 0) 0x7f18dc0848f0 con 0x7f18d80c7a70 2024-03-29T11:25:20.622 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:20.627+0000 7f18d67fc640 10 client.4454.objecter ms_dispatch 0x7f18e0000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:25:20.622 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:20.627+0000 7f18d67fc640 3 client.4454.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:25:20.622 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:20.627+0000 7f18d67fc640 3 client.4454.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:25:20.622 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:20.627+0000 7f18d67fc640 20 client.4454.objecter dump_active .. 0 homeless 2024-03-29T11:25:20.622 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:20.627+0000 7f18d67fc640 10 client.4454.objecter ms_handle_connect 0x7f18c406ded0 2024-03-29T11:25:20.622 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:20.636+0000 7f18d67fc640 1 -- 172.21.15.84:0/4036169308 <== mon.2 v1:172.21.15.84:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f18dc050000 con 0x7f18d80c7a70 2024-03-29T11:25:20.623 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:20.636+0000 7f18d67fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:25:20.623 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:20.636+0000 7f18d67fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:25:20.943 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:20.959+0000 7f18e6cd9640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:25:20.943 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:20.959+0000 7f18e6cd9640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:25:20.943 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:20.959+0000 7f18e6cd9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-29T11:25:20.943 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:20.959+0000 7f18e6cd9640 1 -- 172.21.15.84:0/4036169308 --> v1:172.21.15.84:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f18c8005330 con 0x7f18d80c7a70 2024-03-29T11:25:20.945 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:20.961+0000 7f18d67fc640 1 -- 172.21.15.84:0/4036169308 <== mon.2 v1:172.21.15.84:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4209477549) 0x7f18dc052100 con 0x7f18d80c7a70 2024-03-29T11:25:20.945 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:20.961+0000 7f18d67fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:25:20.945 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:20.961+0000 7f18d67fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:25:20.946 INFO:teuthology.misc.health.smithi084.stdout: 2024-03-29T11:25:20.946 INFO:teuthology.misc.health.smithi084.stdout:{"epoch":10,"fsid":"2c3e002f-9eb0-4860-853b-4ef1ef562213","created":"2024-03-29T11:19:32.029169+0000","modified":"2024-03-29T11:19:34.200299+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:34.200299+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"5c39e276-2034-4716-bbc0-57a634eefa92","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"cd580e99-109f-4bf1-88c0-a5987bc8b9ed","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"11e60a5a-051d-453f-a190-66387cb35592","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"2c329fbe-bbc4-4030-bf87-d711cad04679","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f4213d4a-bd36-4433-91ae-d7e2b02481d3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"f0cf44ea-6055-48af-805b-aed858102ff9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"d895eaa0-cc0b-4ee4-9b5a-2dfcd89a60b7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"9bc5a7bc-8f52-4db9-96d6-de5e68d52eaa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:25:20.949 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:20.965+0000 7f18e6cd9640 10 librados: watch_flush enter 2024-03-29T11:25:20.949 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:20.965+0000 7f18e6cd9640 10 librados: watch_flush exit 2024-03-29T11:25:20.949 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:20.965+0000 7f18e6cd9640 20 client.4454.objecter shutdown clearing up homeless session... 2024-03-29T11:25:20.949 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:20.965+0000 7f18e6cd9640 10 client.4454.objecter successfully canceled tick 2024-03-29T11:25:20.949 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:20.965+0000 7f18e6cd9640 1 -- 172.21.15.84:0/4036169308 >> v1:172.21.15.179:6816/2981134980 conn(0x7f18c406ded0 legacy=0x7f18c4070390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:20.950 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:20.965+0000 7f18e6cd9640 10 monclient: shutdown 2024-03-29T11:25:20.950 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:20.965+0000 7f18e6cd9640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:25:20.950 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:20.965+0000 7f18e6cd9640 1 -- 172.21.15.84:0/4036169308 >> v1:172.21.15.84:6790/0 conn(0x7f18d80c7a70 legacy=0x7f18d802d250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:20.950 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:20.965+0000 7f18e6cd9640 1 -- 172.21.15.84:0/4036169308 shutdown_connections 2024-03-29T11:25:20.950 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:20.965+0000 7f18e6cd9640 1 -- 172.21.15.84:0/4036169308 >> 172.21.15.84:0/4036169308 conn(0x7f18d8005c00 msgr2=0x7f18d8034090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:25:20.950 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:20.965+0000 7f18e6cd9640 1 -- 172.21.15.84:0/4036169308 shutdown_connections 2024-03-29T11:25:20.950 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:20.965+0000 7f18e6cd9640 1 -- 172.21.15.84:0/4036169308 wait complete. 2024-03-29T11:25:20.950 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:20.965+0000 7f18e6cd9640 1 librados: shutdown 2024-03-29T11:25:20.967 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:25:27.774 DEBUG:teuthology.orchestra.run.smithi084:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:25:27.924 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:27.940+0000 7f76911c5640 10 monclient: get_monmap_and_config 2024-03-29T11:25:27.925 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:27.940+0000 7f76911c5640 10 monclient: build_initial_monmap 2024-03-29T11:25:27.925 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:27.940+0000 7f76911c5640 10 monclient: monmap: 2024-03-29T11:25:27.925 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-29T11:25:27.925 INFO:teuthology.misc.health.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:25:27.925 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T11:25:27.941464+0000 2024-03-29T11:25:27.925 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T11:25:27.941464+0000 2024-03-29T11:25:27.925 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-29T11:25:27.925 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:25:27.925 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-29T11:25:27.925 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.noname-c 2024-03-29T11:25:27.925 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-b 2024-03-29T11:25:27.925 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:25:27.926 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:27.942+0000 7f76911c5640 1 Processor -- start 2024-03-29T11:25:27.926 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:27.943+0000 7f76911c5640 1 -- start start 2024-03-29T11:25:27.926 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:27.943+0000 7f76911c5640 10 monclient: init 2024-03-29T11:25:27.927 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:27.943+0000 7f76911c5640 10 monclient: _reopen_session rank -1 2024-03-29T11:25:27.927 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:27.943+0000 7f76911c5640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:25:27.927 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:27.944+0000 7f76911c5640 10 monclient(hunting): picked mon.noname-b con 0x7f768c0b4920 addr v1:172.21.15.84:6790/0 2024-03-29T11:25:27.928 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:27.944+0000 7f768bfff640 1 --1- >> v1:172.21.15.84:6790/0 conn(0x7f768c0b4920 0x7f768c13f290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.84:6790/0 says I am v1:172.21.15.84:37938/0 (socket says 172.21.15.84:37938) 2024-03-29T11:25:27.930 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:27.944+0000 7f768bfff640 1 -- 172.21.15.84:0/1488575776 learned_addr learned my addr 172.21.15.84:0/1488575776 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-29T11:25:27.930 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:27.944+0000 7f76911c5640 10 monclient(hunting): picked mon.noname-c con 0x7f768c0b4200 addr v1:172.21.15.179:6789/0 2024-03-29T11:25:27.930 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:27.944+0000 7f76911c5640 10 monclient(hunting): picked mon.noname-a con 0x7f768c13f9a0 addr v1:172.21.15.84:6789/0 2024-03-29T11:25:27.930 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:27.944+0000 7f76911c5640 1 -- 172.21.15.84:0/1488575776 --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f768c0b0930 con 0x7f768c13f9a0 2024-03-29T11:25:27.930 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:27.944+0000 7f76911c5640 1 -- 172.21.15.84:0/1488575776 --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f768c0af460 con 0x7f768c0b4200 2024-03-29T11:25:27.930 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:27.944+0000 7f76911c5640 1 -- 172.21.15.84:0/1488575776 --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f768c0aeda0 con 0x7f768c0b4920 2024-03-29T11:25:27.930 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:27.944+0000 7f76911c5640 10 monclient(hunting): _renew_subs 2024-03-29T11:25:27.930 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:27.944+0000 7f76911c5640 10 monclient(hunting): authenticate will time out at 1189.238420s 2024-03-29T11:25:27.931 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:27.947+0000 7f76909c4640 1 -- 172.21.15.84:0/1488575776 >> v1:172.21.15.84:6789/0 conn(0x7f768c13f9a0 legacy=0x7f768c14d8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2024-03-29T11:25:27.931 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:27.947+0000 7f76909c4640 1 -- 172.21.15.84:0/1488575776 >> v1:172.21.15.84:6789/0 conn(0x7f768c13f9a0 legacy=0x7f768c14d8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-29T11:25:27.931 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:27.947+0000 7f76909c4640 1 --1- 172.21.15.84:0/1488575776 >> v1:172.21.15.84:6789/0 conn(0x7f768c13f9a0 0x7f768c14d8a0 :-1 s=OPENED pgs=587315 cs=1 l=1).handle_message read tag failed 2024-03-29T11:25:27.932 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:27.947+0000 7f76909c4640 1 --1- 172.21.15.84:0/1488575776 >> v1:172.21.15.84:6789/0 conn(0x7f768c13f9a0 0x7f768c14d8a0 :-1 s=OPENED pgs=587315 cs=1 l=1).fault on lossy channel, failing 2024-03-29T11:25:27.932 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:27.947+0000 7f768affd640 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.84:6789/0 2024-03-29T11:25:27.936 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:27.949+0000 7f768affd640 1 -- 172.21.15.84:0/1488575776 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3896095916 0 0) 0x7f768c0aeda0 con 0x7f768c0b4920 2024-03-29T11:25:27.936 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:27.949+0000 7f768affd640 10 monclient(hunting): _negotiate 2024-03-29T11:25:27.936 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:27.949+0000 7f768affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:25:27.936 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:27.949+0000 7f768affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:25:27.937 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:27.949+0000 7f768affd640 10 monclient(hunting): my global_id is 4457 2024-03-29T11:25:27.937 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:27.949+0000 7f768affd640 1 -- 172.21.15.84:0/1488575776 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7678003520 con 0x7f768c0b4920 2024-03-29T11:25:27.937 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:27.949+0000 7f768affd640 1 -- 172.21.15.84:0/1488575776 <== mon.2 v1:172.21.15.84:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 408734515 0 0) 0x7f7678003520 con 0x7f768c0b4920 2024-03-29T11:25:27.937 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:27.949+0000 7f768affd640 1 -- 172.21.15.84:0/1488575776 >> v1:172.21.15.179:6789/0 conn(0x7f768c0b4200 legacy=0x7f768c0b0fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:25:27.937 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:27.949+0000 7f768affd640 1 -- 172.21.15.84:0/1488575776 >> v1:172.21.15.84:6789/0 conn(0x7f768c13f9a0 legacy=0x7f768c14d8a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-29T11:25:27.937 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:27.949+0000 7f768affd640 10 monclient: _finish_hunting 0 2024-03-29T11:25:27.937 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:27.949+0000 7f768affd640 1 monclient: found mon.noname-b 2024-03-29T11:25:27.937 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:27.949+0000 7f768affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:25:27.937 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:27.949+0000 7f768affd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.84:6790/0 2024-03-29T11:25:27.937 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:27.949+0000 7f768affd640 1 -- 172.21.15.84:0/1488575776 --> v1:172.21.15.84:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f768c0af980 con 0x7f768c0b4920 2024-03-29T11:25:27.937 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:27.949+0000 7f768affd640 10 monclient: _finish_auth 0 2024-03-29T11:25:27.937 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:27.949+0000 7f768affd640 10 monclient: _check_auth_tickets 2024-03-29T11:25:27.938 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:27.949+0000 7f768affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:25:27.938 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:27.949+0000 7f768affd640 1 -- 172.21.15.84:0/1488575776 <== mon.2 v1:172.21.15.84:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4236868694 0 0) 0x7f7684001a10 con 0x7f768c0b4920 2024-03-29T11:25:27.938 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:27.949+0000 7f768affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:25:27.938 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:27.949+0000 7f768affd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:25:27.938 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:27.949+0000 7f768affd640 10 monclient: dump: 2024-03-29T11:25:27.938 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T11:25:27.938 INFO:teuthology.misc.health.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:25:27.938 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T11:18:53.864025+0000 2024-03-29T11:25:27.938 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T11:18:53.864025+0000 2024-03-29T11:25:27.938 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T11:25:27.938 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:25:27.938 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T11:25:27.938 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T11:25:27.938 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T11:25:27.939 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:25:27.939 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:27.949+0000 7f76911c5640 5 monclient: authenticate success, global_id 4457 2024-03-29T11:25:27.939 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:27.949+0000 7f76911c5640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:25:27.939 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:27.949+0000 7f768affd640 1 -- 172.21.15.84:0/1488575776 <== mon.2 v1:172.21.15.84:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7684001b70 con 0x7f768c0b4920 2024-03-29T11:25:27.939 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:27.949+0000 7f768affd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:25:27.939 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:27.949+0000 7f768affd640 1 -- 172.21.15.84:0/1488575776 <== mon.2 v1:172.21.15.84:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4236868694 0 0) 0x7f7684001fc0 con 0x7f768c0b4920 2024-03-29T11:25:27.939 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:27.949+0000 7f768affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:25:27.939 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:27.949+0000 7f768affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:25:27.939 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:27.949+0000 7f768affd640 10 monclient: dump: 2024-03-29T11:25:27.939 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T11:25:27.939 INFO:teuthology.misc.health.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:25:27.939 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T11:18:53.864025+0000 2024-03-29T11:25:27.939 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T11:18:53.864025+0000 2024-03-29T11:25:27.939 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T11:25:27.939 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:25:27.940 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T11:25:27.940 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T11:25:27.940 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T11:25:27.940 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:25:27.940 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:27.949+0000 7f76911c5640 10 monclient: get_monmap_and_config success 2024-03-29T11:25:27.940 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:27.949+0000 7f76911c5640 4 set_mon_vals no callback set 2024-03-29T11:25:27.940 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:27.950+0000 7f76911c5640 10 monclient: shutdown 2024-03-29T11:25:27.940 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:27.950+0000 7f76911c5640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:25:27.940 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:27.950+0000 7f76911c5640 1 -- 172.21.15.84:0/1488575776 >> v1:172.21.15.84:6790/0 conn(0x7f768c0b4920 legacy=0x7f768c13f290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:27.940 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:27.951+0000 7f76911c5640 1 -- 172.21.15.84:0/1488575776 shutdown_connections 2024-03-29T11:25:27.940 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:27.951+0000 7f76911c5640 1 -- 172.21.15.84:0/1488575776 >> 172.21.15.84:0/1488575776 conn(0x7f768c06d230 msgr2=0x7f768c066a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:25:27.940 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:27.953+0000 7f76911c5640 1 -- 172.21.15.84:0/1488575776 shutdown_connections 2024-03-29T11:25:27.940 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:27.953+0000 7f76911c5640 1 -- 172.21.15.84:0/1488575776 wait complete. 2024-03-29T11:25:27.942 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:27.953+0000 7f76911c5640 10 monclient: build_initial_monmap 2024-03-29T11:25:27.942 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:27.953+0000 7f76911c5640 10 monclient: monmap: 2024-03-29T11:25:27.942 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-29T11:25:27.942 INFO:teuthology.misc.health.smithi084.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:25:27.942 INFO:teuthology.misc.health.smithi084.stderr:last_changed 0.000000 2024-03-29T11:25:27.942 INFO:teuthology.misc.health.smithi084.stderr:created 0.000000 2024-03-29T11:25:27.942 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-29T11:25:27.942 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:25:27.942 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-29T11:25:27.942 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.noname-b 2024-03-29T11:25:27.942 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-c 2024-03-29T11:25:27.943 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:25:27.943 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:27.954+0000 7f76911c5640 1 librados: starting msgr at 2024-03-29T11:25:27.943 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:27.954+0000 7f76911c5640 1 librados: starting objecter 2024-03-29T11:25:27.943 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:27.954+0000 7f76911c5640 1 Processor -- start 2024-03-29T11:25:27.943 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:27.954+0000 7f76911c5640 1 -- start start 2024-03-29T11:25:27.943 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:27.954+0000 7f76911c5640 1 librados: setting wanted keys 2024-03-29T11:25:27.943 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:27.954+0000 7f76911c5640 1 librados: calling monclient init 2024-03-29T11:25:27.943 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:27.954+0000 7f76911c5640 10 monclient: init 2024-03-29T11:25:27.943 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:27.955+0000 7f76911c5640 10 monclient: _reopen_session rank -1 2024-03-29T11:25:27.943 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:27.955+0000 7f76911c5640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:25:27.943 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:27.955+0000 7f76911c5640 10 monclient(hunting): picked mon.noname-c con 0x7f768c0b4200 addr v1:172.21.15.84:6790/0 2024-03-29T11:25:27.943 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:27.955+0000 7f76911c5640 10 monclient(hunting): picked mon.noname-b con 0x7f768c0b4920 addr v1:172.21.15.179:6789/0 2024-03-29T11:25:27.943 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:27.955+0000 7f76911c5640 10 monclient(hunting): picked mon.noname-a con 0x7f768c13f9a0 addr v1:172.21.15.84:6789/0 2024-03-29T11:25:27.943 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:27.955+0000 7f76911c5640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f768c145ac0 con 0x7f768c13f9a0 2024-03-29T11:25:27.944 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:27.955+0000 7f76911c5640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f768c1457a0 con 0x7f768c0b4920 2024-03-29T11:25:27.944 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:27.955+0000 7f76911c5640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f768c145f50 con 0x7f768c0b4200 2024-03-29T11:25:27.944 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:27.955+0000 7f76911c5640 10 monclient(hunting): _renew_subs 2024-03-29T11:25:27.944 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:27.955+0000 7f76911c5640 10 monclient(hunting): authenticate will time out at 1189.249755s 2024-03-29T11:25:27.944 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:27.955+0000 7f768bfff640 1 --1- >> v1:172.21.15.84:6790/0 conn(0x7f768c0b4200 0x7f768c145090 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.84:6790/0 says I am v1:172.21.15.84:39014/0 (socket says 172.21.15.84:39014) 2024-03-29T11:25:27.944 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:27.955+0000 7f768bfff640 1 -- 172.21.15.84:0/124929872 learned_addr learned my addr 172.21.15.84:0/124929872 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-29T11:25:27.944 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:27.956+0000 7f7688ff9640 10 client.?.objecter ms_handle_connect 0x7f768c0b4200 2024-03-29T11:25:27.944 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:27.956+0000 7f7688ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:25:27.944 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:27.956+0000 7f7688ff9640 1 -- 172.21.15.84:0/124929872 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 636388273 0 0) 0x7f768c145f50 con 0x7f768c0b4200 2024-03-29T11:25:27.944 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:27.956+0000 7f7688ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:25:27.944 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:27.956+0000 7f7688ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:25:27.944 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:27.956+0000 7f7688ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:25:27.944 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:27.956+0000 7f7688ff9640 10 monclient(hunting): my global_id is 4460 2024-03-29T11:25:27.944 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:27.956+0000 7f7688ff9640 1 -- 172.21.15.84:0/124929872 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f766c004eb0 con 0x7f768c0b4200 2024-03-29T11:25:27.944 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:27.957+0000 7f7688ff9640 1 -- 172.21.15.84:0/124929872 <== mon.2 v1:172.21.15.84:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3937295155 0 0) 0x7f766c004eb0 con 0x7f768c0b4200 2024-03-29T11:25:27.945 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:27.957+0000 7f7688ff9640 1 -- 172.21.15.84:0/124929872 --> v1:172.21.15.84:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f768c145f50 con 0x7f768c0b4200 2024-03-29T11:25:27.945 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:27.957+0000 7f7688ff9640 1 -- 172.21.15.84:0/124929872 <== mon.2 v1:172.21.15.84:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4236868694 0 0) 0x7f7684001e80 con 0x7f768c0b4200 2024-03-29T11:25:27.945 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:27.957+0000 7f7688ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:25:27.945 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:27.957+0000 7f7688ff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:25:27.945 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:27.957+0000 7f7688ff9640 10 monclient(hunting): dump: 2024-03-29T11:25:27.945 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T11:25:27.945 INFO:teuthology.misc.health.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:25:27.945 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T11:18:53.864025+0000 2024-03-29T11:25:27.945 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T11:18:53.864025+0000 2024-03-29T11:25:27.945 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T11:25:27.945 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:25:27.945 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T11:25:27.945 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T11:25:27.945 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T11:25:27.945 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:25:27.946 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:27.957+0000 7f7688ff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:25:27.946 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:27.957+0000 7f7688ff9640 1 -- 172.21.15.84:0/124929872 <== mon.2 v1:172.21.15.84:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1036898013 0 0) 0x7f768c145f50 con 0x7f768c0b4200 2024-03-29T11:25:27.946 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:27.957+0000 7f7688ff9640 1 -- 172.21.15.84:0/124929872 >> v1:172.21.15.179:6789/0 conn(0x7f768c0b4920 legacy=0x7f768c06ef40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:25:27.949 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:27.961+0000 7f7688ff9640 1 -- 172.21.15.84:0/124929872 >> v1:172.21.15.84:6789/0 conn(0x7f768c13f9a0 legacy=0x7f768c1ff960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:27.949 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:27.961+0000 7f7688ff9640 10 monclient: _finish_hunting 0 2024-03-29T11:25:27.949 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:27.961+0000 7f7688ff9640 1 monclient: found mon.c 2024-03-29T11:25:27.949 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:27.961+0000 7f7688ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:25:27.949 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:27.961+0000 7f7688ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-29T11:25:27.949 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:27.961+0000 7f7688ff9640 1 -- 172.21.15.84:0/124929872 --> v1:172.21.15.84:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f768c1462e0 con 0x7f768c0b4200 2024-03-29T11:25:27.949 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:27.961+0000 7f7688ff9640 10 monclient: _finish_auth 0 2024-03-29T11:25:27.949 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:27.962+0000 7f7688ff9640 10 monclient: _check_auth_tickets 2024-03-29T11:25:27.949 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:27.962+0000 7f7688ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:25:27.949 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:27.962+0000 7f7688ff9640 10 client.?.objecter ms_handle_connect 0x7f768c13f9a0 2024-03-29T11:25:27.950 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:27.962+0000 7f7688ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:25:27.950 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:27.962+0000 7f7688ff9640 1 -- 172.21.15.84:0/124929872 <== mon.2 v1:172.21.15.84:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7684004f70 con 0x7f768c0b4200 2024-03-29T11:25:27.950 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:27.962+0000 7f76911c5640 5 monclient: authenticate success, global_id 4460 2024-03-29T11:25:27.950 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:27.962+0000 7f76911c5640 10 monclient: _renew_subs 2024-03-29T11:25:27.950 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:27.962+0000 7f76911c5640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-29T11:25:27.950 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:27.962+0000 7f76911c5640 1 -- 172.21.15.84:0/124929872 --> v1:172.21.15.84:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f768c06f650 con 0x7f768c0b4200 2024-03-29T11:25:27.950 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:27.962+0000 7f76911c5640 10 client.4460.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:25:27.950 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:27.962+0000 7f76911c5640 10 monclient: _renew_subs 2024-03-29T11:25:27.950 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:27.962+0000 7f76911c5640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-29T11:25:27.950 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:27.962+0000 7f76911c5640 1 -- 172.21.15.84:0/124929872 --> v1:172.21.15.84:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f768c142410 con 0x7f768c0b4200 2024-03-29T11:25:27.950 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:27.962+0000 7f76911c5640 1 librados: init done 2024-03-29T11:25:27.950 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:27.963+0000 7f7688ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:25:27.950 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:27.963+0000 7f7688ff9640 1 -- 172.21.15.84:0/124929872 <== mon.2 v1:172.21.15.84:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4236868694 0 0) 0x7f7684005280 con 0x7f768c0b4200 2024-03-29T11:25:27.950 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:27.963+0000 7f7689ffb640 4 set_mon_vals no callback set 2024-03-29T11:25:27.951 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:27.963+0000 7f7688ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:25:27.951 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:27.963+0000 7f7688ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:25:27.951 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:27.963+0000 7f7688ff9640 10 monclient: dump: 2024-03-29T11:25:27.951 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T11:25:27.951 INFO:teuthology.misc.health.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:25:27.951 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T11:18:53.864025+0000 2024-03-29T11:25:27.951 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T11:18:53.864025+0000 2024-03-29T11:25:27.951 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T11:25:27.951 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:25:27.951 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T11:25:27.951 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T11:25:27.951 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T11:25:27.951 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:25:27.951 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:27.964+0000 7f76911c5640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:25:27.951 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:27.964+0000 7f76911c5640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:25:27.952 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:27.964+0000 7f76911c5640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-29T11:25:27.952 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:27.964+0000 7f76911c5640 1 -- 172.21.15.84:0/124929872 --> v1:172.21.15.84:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7670005080 con 0x7f768c0b4200 2024-03-29T11:25:27.953 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:27.966+0000 7f7688ff9640 1 -- 172.21.15.84:0/124929872 <== mon.2 v1:172.21.15.84:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4033942909 0 0) 0x7f768401b250 con 0x7f768c0b4200 2024-03-29T11:25:27.953 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:27.967+0000 7f7688ff9640 1 -- 172.21.15.84:0/124929872 <== mon.2 v1:172.21.15.84:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2808139587 0 0) 0x7f7684084fd0 con 0x7f768c0b4200 2024-03-29T11:25:27.953 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:27.967+0000 7f7688ff9640 10 client.4460.objecter ms_dispatch 0x7f768c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:25:27.953 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:27.967+0000 7f7688ff9640 3 client.4460.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:25:27.953 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:27.967+0000 7f768b7fe640 10 monclient: get_auth_request con 0x7f766c06dcf0 auth_method 0 2024-03-29T11:25:27.953 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:27.967+0000 7f7688ff9640 3 client.4460.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:25:27.953 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:27.967+0000 7f7688ff9640 20 client.4460.objecter dump_active .. 0 homeless 2024-03-29T11:25:27.953 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:27.967+0000 7f7688ff9640 10 client.4460.objecter ms_handle_connect 0x7f766c06dcf0 2024-03-29T11:25:27.957 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:27.973+0000 7f7688ff9640 1 -- 172.21.15.84:0/124929872 <== mon.2 v1:172.21.15.84:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f76840508b0 con 0x7f768c0b4200 2024-03-29T11:25:27.957 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:27.973+0000 7f7688ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:25:27.957 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:27.973+0000 7f7688ff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:25:28.285 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:28.301+0000 7f76911c5640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:25:28.288 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:28.301+0000 7f76911c5640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:25:28.288 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:28.301+0000 7f76911c5640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-29T11:25:28.288 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:28.301+0000 7f76911c5640 1 -- 172.21.15.84:0/124929872 --> v1:172.21.15.84:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7670005330 con 0x7f768c0b4200 2024-03-29T11:25:28.288 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:28.304+0000 7f7688ff9640 1 -- 172.21.15.84:0/124929872 <== mon.2 v1:172.21.15.84:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4209477549) 0x7f76840527e0 con 0x7f768c0b4200 2024-03-29T11:25:28.288 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:28.304+0000 7f7688ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:25:28.288 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:28.304+0000 7f7688ff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:25:28.290 INFO:teuthology.misc.health.smithi084.stdout: 2024-03-29T11:25:28.291 INFO:teuthology.misc.health.smithi084.stdout:{"epoch":10,"fsid":"2c3e002f-9eb0-4860-853b-4ef1ef562213","created":"2024-03-29T11:19:32.029169+0000","modified":"2024-03-29T11:19:34.200299+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:34.200299+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"5c39e276-2034-4716-bbc0-57a634eefa92","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"cd580e99-109f-4bf1-88c0-a5987bc8b9ed","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"11e60a5a-051d-453f-a190-66387cb35592","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"2c329fbe-bbc4-4030-bf87-d711cad04679","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f4213d4a-bd36-4433-91ae-d7e2b02481d3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"f0cf44ea-6055-48af-805b-aed858102ff9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"d895eaa0-cc0b-4ee4-9b5a-2dfcd89a60b7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"9bc5a7bc-8f52-4db9-96d6-de5e68d52eaa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:25:28.294 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:28.309+0000 7f76911c5640 10 librados: watch_flush enter 2024-03-29T11:25:28.294 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:28.309+0000 7f76911c5640 10 librados: watch_flush exit 2024-03-29T11:25:28.294 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:28.309+0000 7f76911c5640 20 client.4460.objecter shutdown clearing up homeless session... 2024-03-29T11:25:28.294 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:28.309+0000 7f76911c5640 10 client.4460.objecter successfully canceled tick 2024-03-29T11:25:28.295 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:28.311+0000 7f76911c5640 1 -- 172.21.15.84:0/124929872 >> v1:172.21.15.179:6816/2981134980 conn(0x7f766c06dcf0 legacy=0x7f766c0701b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:28.295 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:28.311+0000 7f76911c5640 10 monclient: shutdown 2024-03-29T11:25:28.295 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:28.311+0000 7f76911c5640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:25:28.295 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:28.311+0000 7f76911c5640 1 -- 172.21.15.84:0/124929872 >> v1:172.21.15.84:6790/0 conn(0x7f768c0b4200 legacy=0x7f768c145090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:28.296 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:28.311+0000 7f76911c5640 1 -- 172.21.15.84:0/124929872 shutdown_connections 2024-03-29T11:25:28.296 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:28.311+0000 7f76911c5640 1 -- 172.21.15.84:0/124929872 >> 172.21.15.84:0/124929872 conn(0x7f768c06d230 msgr2=0x7f768c0b31b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:25:28.296 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:28.312+0000 7f76911c5640 1 -- 172.21.15.84:0/124929872 shutdown_connections 2024-03-29T11:25:28.296 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:28.312+0000 7f76911c5640 1 -- 172.21.15.84:0/124929872 wait complete. 2024-03-29T11:25:28.296 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:28.312+0000 7f76911c5640 1 librados: shutdown 2024-03-29T11:25:28.314 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:25:35.121 DEBUG:teuthology.orchestra.run.smithi084:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:25:35.268 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:35.283+0000 7fb2a648f640 10 monclient: get_monmap_and_config 2024-03-29T11:25:35.268 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:35.283+0000 7fb2a648f640 10 monclient: build_initial_monmap 2024-03-29T11:25:35.268 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:35.283+0000 7fb2a648f640 10 monclient: monmap: 2024-03-29T11:25:35.268 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-29T11:25:35.268 INFO:teuthology.misc.health.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:25:35.268 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T11:25:35.284757+0000 2024-03-29T11:25:35.268 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T11:25:35.284757+0000 2024-03-29T11:25:35.268 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-29T11:25:35.268 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:25:35.269 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-29T11:25:35.269 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.noname-c 2024-03-29T11:25:35.269 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-b 2024-03-29T11:25:35.269 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:25:35.270 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:35.286+0000 7fb2a648f640 1 Processor -- start 2024-03-29T11:25:35.270 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:35.286+0000 7fb2a648f640 1 -- start start 2024-03-29T11:25:35.270 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:35.286+0000 7fb2a648f640 10 monclient: init 2024-03-29T11:25:35.272 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:35.286+0000 7fb2a648f640 10 monclient: _reopen_session rank -1 2024-03-29T11:25:35.272 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:35.286+0000 7fb2a648f640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:25:35.272 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:35.287+0000 7fb2a648f640 10 monclient(hunting): picked mon.noname-c con 0x7fb2a01457f0 addr v1:172.21.15.179:6789/0 2024-03-29T11:25:35.272 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:35.287+0000 7fb2a648f640 10 monclient(hunting): picked mon.noname-a con 0x7fb2a013f650 addr v1:172.21.15.84:6789/0 2024-03-29T11:25:35.272 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:35.287+0000 7fb2a648f640 10 monclient(hunting): picked mon.noname-b con 0x7fb2a00a9d70 addr v1:172.21.15.84:6790/0 2024-03-29T11:25:35.272 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:35.287+0000 7fb2a648f640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2a00aaef0 con 0x7fb2a013f650 2024-03-29T11:25:35.272 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:35.287+0000 7fb2a648f640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2a0145bd0 con 0x7fb2a01457f0 2024-03-29T11:25:35.272 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:35.287+0000 7fb2a648f640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2a0145eb0 con 0x7fb2a00a9d70 2024-03-29T11:25:35.272 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:35.287+0000 7fb2a648f640 10 monclient(hunting): _renew_subs 2024-03-29T11:25:35.272 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:35.287+0000 7fb2a648f640 10 monclient(hunting): authenticate will time out at 1196.581192s 2024-03-29T11:25:35.272 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:35.287+0000 7fb2a4a2c640 1 --1- >> v1:172.21.15.84:6790/0 conn(0x7fb2a00a9d70 0x7fb2a014d8c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.84:6790/0 says I am v1:172.21.15.84:54652/0 (socket says 172.21.15.84:54652) 2024-03-29T11:25:35.272 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:35.287+0000 7fb2a4a2c640 1 -- 172.21.15.84:0/1695953790 learned_addr learned my addr 172.21.15.84:0/1695953790 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-29T11:25:35.273 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:35.287+0000 7fb29effd640 1 -- 172.21.15.84:0/1695953790 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1955895351 0 0) 0x7fb2a0145eb0 con 0x7fb2a00a9d70 2024-03-29T11:25:35.273 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:35.287+0000 7fb29effd640 10 monclient(hunting): _negotiate 2024-03-29T11:25:35.273 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:35.287+0000 7fb29effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:25:35.273 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:35.287+0000 7fb29effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:25:35.273 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:35.287+0000 7fb29effd640 10 monclient(hunting): my global_id is 4463 2024-03-29T11:25:35.273 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:35.287+0000 7fb29effd640 1 -- 172.21.15.84:0/1695953790 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb288003520 con 0x7fb2a00a9d70 2024-03-29T11:25:35.274 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:35.290+0000 7fb29effd640 1 -- 172.21.15.84:0/1695953790 <== mon.2 v1:172.21.15.84:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 418395209 0 0) 0x7fb288003520 con 0x7fb2a00a9d70 2024-03-29T11:25:35.274 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:35.290+0000 7fb29effd640 1 -- 172.21.15.84:0/1695953790 >> v1:172.21.15.179:6789/0 conn(0x7fb2a01457f0 legacy=0x7fb2a013ef40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:25:35.275 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:35.290+0000 7fb29effd640 1 -- 172.21.15.84:0/1695953790 >> v1:172.21.15.84:6789/0 conn(0x7fb2a013f650 legacy=0x7fb2a00b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:25:35.275 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:35.290+0000 7fb29effd640 10 monclient: _finish_hunting 0 2024-03-29T11:25:35.275 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:35.290+0000 7fb29effd640 1 monclient: found mon.noname-b 2024-03-29T11:25:35.275 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:35.290+0000 7fb29effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:25:35.275 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:35.290+0000 7fb29effd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.84:6790/0 2024-03-29T11:25:35.275 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:35.290+0000 7fb29effd640 1 -- 172.21.15.84:0/1695953790 --> v1:172.21.15.84:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2a0146340 con 0x7fb2a00a9d70 2024-03-29T11:25:35.275 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:35.290+0000 7fb29effd640 10 monclient: _finish_auth 0 2024-03-29T11:25:35.275 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:35.290+0000 7fb29effd640 10 monclient: _check_auth_tickets 2024-03-29T11:25:35.275 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:35.290+0000 7fb29effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:25:35.275 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:35.290+0000 7fb29effd640 1 -- 172.21.15.84:0/1695953790 <== mon.2 v1:172.21.15.84:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4236868694 0 0) 0x7fb298001a10 con 0x7fb2a00a9d70 2024-03-29T11:25:35.276 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:35.290+0000 7fb29effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:25:35.276 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:35.290+0000 7fb29effd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:25:35.276 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:35.290+0000 7fb29effd640 10 monclient: dump: 2024-03-29T11:25:35.276 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T11:25:35.276 INFO:teuthology.misc.health.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:25:35.276 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T11:18:53.864025+0000 2024-03-29T11:25:35.276 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T11:18:53.864025+0000 2024-03-29T11:25:35.276 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T11:25:35.276 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:25:35.276 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T11:25:35.276 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T11:25:35.276 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T11:25:35.276 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:25:35.277 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:35.290+0000 7fb2a648f640 5 monclient: authenticate success, global_id 4463 2024-03-29T11:25:35.277 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:35.290+0000 7fb2a648f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:25:35.277 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:35.290+0000 7fb29effd640 1 -- 172.21.15.84:0/1695953790 <== mon.2 v1:172.21.15.84:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb298001be0 con 0x7fb2a00a9d70 2024-03-29T11:25:35.277 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:35.290+0000 7fb29effd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:25:35.277 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:35.290+0000 7fb2a648f640 10 monclient: get_monmap_and_config success 2024-03-29T11:25:35.277 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:35.290+0000 7fb2a648f640 4 set_mon_vals no callback set 2024-03-29T11:25:35.277 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:35.290+0000 7fb29effd640 1 -- 172.21.15.84:0/1695953790 <== mon.2 v1:172.21.15.84:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4236868694 0 0) 0x7fb298002030 con 0x7fb2a00a9d70 2024-03-29T11:25:35.277 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:35.290+0000 7fb29effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:25:35.277 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:35.290+0000 7fb29effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:25:35.277 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:35.290+0000 7fb29effd640 10 monclient: dump: 2024-03-29T11:25:35.277 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T11:25:35.277 INFO:teuthology.misc.health.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:25:35.277 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T11:18:53.864025+0000 2024-03-29T11:25:35.277 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T11:18:53.864025+0000 2024-03-29T11:25:35.277 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T11:25:35.278 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:25:35.278 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T11:25:35.278 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T11:25:35.278 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T11:25:35.278 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:25:35.285 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:35.291+0000 7fb2a648f640 10 monclient: shutdown 2024-03-29T11:25:35.285 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:35.291+0000 7fb2a648f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:25:35.285 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:35.291+0000 7fb2a648f640 1 -- 172.21.15.84:0/1695953790 >> v1:172.21.15.84:6790/0 conn(0x7fb2a00a9d70 legacy=0x7fb2a014d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:35.285 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:35.291+0000 7fb2a648f640 1 -- 172.21.15.84:0/1695953790 shutdown_connections 2024-03-29T11:25:35.285 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:35.291+0000 7fb2a648f640 1 -- 172.21.15.84:0/1695953790 >> 172.21.15.84:0/1695953790 conn(0x7fb2a0066a00 msgr2=0x7fb2a0066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:25:35.285 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:35.292+0000 7fb2a648f640 1 -- 172.21.15.84:0/1695953790 shutdown_connections 2024-03-29T11:25:35.285 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:35.292+0000 7fb2a648f640 1 -- 172.21.15.84:0/1695953790 wait complete. 2024-03-29T11:25:35.285 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:35.293+0000 7fb2a648f640 10 monclient: build_initial_monmap 2024-03-29T11:25:35.285 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:35.293+0000 7fb2a648f640 10 monclient: monmap: 2024-03-29T11:25:35.286 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-29T11:25:35.286 INFO:teuthology.misc.health.smithi084.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:25:35.286 INFO:teuthology.misc.health.smithi084.stderr:last_changed 0.000000 2024-03-29T11:25:35.286 INFO:teuthology.misc.health.smithi084.stderr:created 0.000000 2024-03-29T11:25:35.286 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-29T11:25:35.286 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:25:35.286 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-29T11:25:35.286 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.noname-b 2024-03-29T11:25:35.286 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-c 2024-03-29T11:25:35.286 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:25:35.286 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:35.294+0000 7fb2a648f640 1 librados: starting msgr at 2024-03-29T11:25:35.286 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:35.294+0000 7fb2a648f640 1 librados: starting objecter 2024-03-29T11:25:35.286 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:35.294+0000 7fb2a648f640 1 Processor -- start 2024-03-29T11:25:35.287 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:35.294+0000 7fb2a648f640 1 -- start start 2024-03-29T11:25:35.287 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:35.294+0000 7fb2a648f640 1 librados: setting wanted keys 2024-03-29T11:25:35.287 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:35.294+0000 7fb2a648f640 1 librados: calling monclient init 2024-03-29T11:25:35.287 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:35.294+0000 7fb2a648f640 10 monclient: init 2024-03-29T11:25:35.287 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:35.295+0000 7fb2a648f640 10 monclient: _reopen_session rank -1 2024-03-29T11:25:35.287 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:35.295+0000 7fb2a648f640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:25:35.287 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:35.295+0000 7fb2a648f640 10 monclient(hunting): picked mon.noname-a con 0x7fb2a00a9d70 addr v1:172.21.15.84:6789/0 2024-03-29T11:25:35.287 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:35.295+0000 7fb2a648f640 10 monclient(hunting): picked mon.noname-b con 0x7fb2a013f650 addr v1:172.21.15.179:6789/0 2024-03-29T11:25:35.287 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:35.295+0000 7fb2a648f640 10 monclient(hunting): picked mon.noname-c con 0x7fb2a01457f0 addr v1:172.21.15.84:6790/0 2024-03-29T11:25:35.287 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:35.295+0000 7fb2a648f640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2a0142300 con 0x7fb2a00a9d70 2024-03-29T11:25:35.287 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:35.295+0000 7fb2a648f640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2a006de90 con 0x7fb2a013f650 2024-03-29T11:25:35.287 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:35.295+0000 7fb2a648f640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2a006e020 con 0x7fb2a01457f0 2024-03-29T11:25:35.287 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:35.295+0000 7fb2a648f640 10 monclient(hunting): _renew_subs 2024-03-29T11:25:35.287 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:35.295+0000 7fb2a648f640 10 monclient(hunting): authenticate will time out at 1196.589539s 2024-03-29T11:25:35.288 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:35.296+0000 7fb2a4a2c640 1 --1- >> v1:172.21.15.84:6790/0 conn(0x7fb2a01457f0 0x7fb2a01e5eb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.84:6790/0 says I am v1:172.21.15.84:55726/0 (socket says 172.21.15.84:55726) 2024-03-29T11:25:35.288 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:35.296+0000 7fb2a4a2c640 1 -- 172.21.15.84:0/1356351676 learned_addr learned my addr 172.21.15.84:0/1356351676 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-29T11:25:35.288 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:35.296+0000 7fb29cff9640 10 client.?.objecter ms_handle_connect 0x7fb2a01457f0 2024-03-29T11:25:35.288 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:35.296+0000 7fb29cff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:25:35.288 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:35.297+0000 7fb29cff9640 1 -- 172.21.15.84:0/1356351676 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4210524417 0 0) 0x7fb2a006e020 con 0x7fb2a01457f0 2024-03-29T11:25:35.288 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:35.297+0000 7fb29cff9640 10 monclient(hunting): _negotiate 2024-03-29T11:25:35.288 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:35.297+0000 7fb29cff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:25:35.288 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:35.297+0000 7fb29cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:25:35.288 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:35.297+0000 7fb29cff9640 10 monclient(hunting): my global_id is 4466 2024-03-29T11:25:35.288 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:35.297+0000 7fb29cff9640 1 -- 172.21.15.84:0/1356351676 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb28c004eb0 con 0x7fb2a01457f0 2024-03-29T11:25:35.288 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:35.297+0000 7fb29cff9640 1 -- 172.21.15.84:0/1356351676 <== mon.2 v1:172.21.15.84:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3022695736 0 0) 0x7fb28c004eb0 con 0x7fb2a01457f0 2024-03-29T11:25:35.288 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:35.297+0000 7fb29cff9640 1 -- 172.21.15.84:0/1356351676 --> v1:172.21.15.84:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb2a006e020 con 0x7fb2a01457f0 2024-03-29T11:25:35.288 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:35.297+0000 7fb29cff9640 1 -- 172.21.15.84:0/1356351676 <== mon.2 v1:172.21.15.84:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4236868694 0 0) 0x7fb298001ef0 con 0x7fb2a01457f0 2024-03-29T11:25:35.289 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:35.297+0000 7fb29cff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:25:35.289 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:35.297+0000 7fb29cff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:25:35.289 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:35.297+0000 7fb29cff9640 10 monclient(hunting): dump: 2024-03-29T11:25:35.289 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T11:25:35.289 INFO:teuthology.misc.health.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:25:35.289 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T11:18:53.864025+0000 2024-03-29T11:25:35.289 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T11:18:53.864025+0000 2024-03-29T11:25:35.289 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T11:25:35.289 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:25:35.289 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T11:25:35.289 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T11:25:35.289 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T11:25:35.289 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:25:35.289 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:35.297+0000 7fb29cff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:25:35.289 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:35.298+0000 7fb29cff9640 1 -- 172.21.15.84:0/1356351676 <== mon.2 v1:172.21.15.84:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 328213869 0 0) 0x7fb2a006e020 con 0x7fb2a01457f0 2024-03-29T11:25:35.290 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:35.298+0000 7fb29cff9640 1 -- 172.21.15.84:0/1356351676 >> v1:172.21.15.179:6789/0 conn(0x7fb2a013f650 legacy=0x7fb2a0142820 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:25:35.290 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:35.299+0000 7fb29cff9640 1 -- 172.21.15.84:0/1356351676 >> v1:172.21.15.84:6789/0 conn(0x7fb2a00a9d70 legacy=0x7fb2a0141aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:35.290 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:35.299+0000 7fb29cff9640 10 monclient: _finish_hunting 0 2024-03-29T11:25:35.290 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:35.299+0000 7fb29cff9640 1 monclient: found mon.c 2024-03-29T11:25:35.290 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:35.299+0000 7fb29cff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:25:35.290 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:35.299+0000 7fb29cff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-29T11:25:35.290 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:35.299+0000 7fb29cff9640 1 -- 172.21.15.84:0/1356351676 --> v1:172.21.15.84:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2a006e1b0 con 0x7fb2a01457f0 2024-03-29T11:25:35.290 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:35.299+0000 7fb29cff9640 10 monclient: _finish_auth 0 2024-03-29T11:25:35.290 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:35.299+0000 7fb29cff9640 10 monclient: _check_auth_tickets 2024-03-29T11:25:35.290 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:35.299+0000 7fb29cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:25:35.290 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:35.299+0000 7fb2a648f640 5 monclient: authenticate success, global_id 4466 2024-03-29T11:25:35.290 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:35.299+0000 7fb2a648f640 10 monclient: _renew_subs 2024-03-29T11:25:35.290 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:35.299+0000 7fb2a648f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-29T11:25:35.290 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:35.299+0000 7fb2a648f640 1 -- 172.21.15.84:0/1356351676 --> v1:172.21.15.84:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb2a013b2a0 con 0x7fb2a01457f0 2024-03-29T11:25:35.291 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:35.299+0000 7fb2a648f640 10 client.4466.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:25:35.291 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:35.299+0000 7fb2a648f640 10 monclient: _renew_subs 2024-03-29T11:25:35.291 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:35.299+0000 7fb2a648f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-29T11:25:35.291 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:35.299+0000 7fb2a648f640 1 -- 172.21.15.84:0/1356351676 --> v1:172.21.15.84:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb2a013fda0 con 0x7fb2a01457f0 2024-03-29T11:25:35.291 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:35.299+0000 7fb2a648f640 1 librados: init done 2024-03-29T11:25:35.293 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:35.301+0000 7fb29cff9640 1 -- 172.21.15.84:0/1356351676 <== mon.2 v1:172.21.15.84:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb298004f70 con 0x7fb2a01457f0 2024-03-29T11:25:35.293 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:35.301+0000 7fb29cff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:25:35.293 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:35.301+0000 7fb29cff9640 1 -- 172.21.15.84:0/1356351676 <== mon.2 v1:172.21.15.84:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4236868694 0 0) 0x7fb298005280 con 0x7fb2a01457f0 2024-03-29T11:25:35.293 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:35.301+0000 7fb29cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:25:35.293 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:35.301+0000 7fb29cff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:25:35.293 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:35.301+0000 7fb29cff9640 10 monclient: dump: 2024-03-29T11:25:35.293 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T11:25:35.293 INFO:teuthology.misc.health.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:25:35.293 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T11:18:53.864025+0000 2024-03-29T11:25:35.294 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T11:18:53.864025+0000 2024-03-29T11:25:35.294 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T11:25:35.294 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:25:35.294 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T11:25:35.294 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T11:25:35.294 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T11:25:35.294 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:25:35.294 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:35.301+0000 7fb29e7fc640 4 set_mon_vals no callback set 2024-03-29T11:25:35.294 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:35.303+0000 7fb2a648f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:25:35.294 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:35.303+0000 7fb2a648f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:25:35.294 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:35.303+0000 7fb2a648f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-29T11:25:35.294 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:35.303+0000 7fb2a648f640 1 -- 172.21.15.84:0/1356351676 --> v1:172.21.15.84:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb280005080 con 0x7fb2a01457f0 2024-03-29T11:25:35.294 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:35.303+0000 7fb29cff9640 1 -- 172.21.15.84:0/1356351676 <== mon.2 v1:172.21.15.84:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4033942909 0 0) 0x7fb29801b250 con 0x7fb2a01457f0 2024-03-29T11:25:35.294 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:35.304+0000 7fb29cff9640 1 -- 172.21.15.84:0/1356351676 <== mon.2 v1:172.21.15.84:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2808139587 0 0) 0x7fb298085060 con 0x7fb2a01457f0 2024-03-29T11:25:35.295 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:35.304+0000 7fb29cff9640 10 client.4466.objecter ms_dispatch 0x7fb2a0000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:25:35.295 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:35.304+0000 7fb29cff9640 3 client.4466.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:25:35.295 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:35.304+0000 7fb29cff9640 3 client.4466.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:25:35.295 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:35.304+0000 7fb29cff9640 20 client.4466.objecter dump_active .. 0 homeless 2024-03-29T11:25:35.295 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:35.304+0000 7fb29ffff640 10 monclient: get_auth_request con 0x7fb28c06dc60 auth_method 0 2024-03-29T11:25:35.295 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:35.306+0000 7fb29cff9640 10 client.4466.objecter ms_handle_connect 0x7fb28c06dc60 2024-03-29T11:25:35.295 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:35.311+0000 7fb29cff9640 1 -- 172.21.15.84:0/1356351676 <== mon.2 v1:172.21.15.84:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fb298050940 con 0x7fb2a01457f0 2024-03-29T11:25:35.296 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:35.311+0000 7fb29cff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:25:35.296 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:35.311+0000 7fb29cff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:25:35.624 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:35.639+0000 7fb2a648f640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:25:35.624 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:35.639+0000 7fb2a648f640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:25:35.624 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:35.639+0000 7fb2a648f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-29T11:25:35.624 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:35.640+0000 7fb2a648f640 1 -- 172.21.15.84:0/1356351676 --> v1:172.21.15.84:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb280005330 con 0x7fb2a01457f0 2024-03-29T11:25:35.625 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:35.641+0000 7fb29cff9640 1 -- 172.21.15.84:0/1356351676 <== mon.2 v1:172.21.15.84:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4209477549) 0x7fb298052870 con 0x7fb2a01457f0 2024-03-29T11:25:35.625 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:35.641+0000 7fb29cff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:25:35.625 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:35.641+0000 7fb29cff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:25:35.625 INFO:teuthology.misc.health.smithi084.stdout: 2024-03-29T11:25:35.626 INFO:teuthology.misc.health.smithi084.stdout:{"epoch":10,"fsid":"2c3e002f-9eb0-4860-853b-4ef1ef562213","created":"2024-03-29T11:19:32.029169+0000","modified":"2024-03-29T11:19:34.200299+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:34.200299+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"5c39e276-2034-4716-bbc0-57a634eefa92","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"cd580e99-109f-4bf1-88c0-a5987bc8b9ed","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"11e60a5a-051d-453f-a190-66387cb35592","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"2c329fbe-bbc4-4030-bf87-d711cad04679","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f4213d4a-bd36-4433-91ae-d7e2b02481d3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"f0cf44ea-6055-48af-805b-aed858102ff9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"d895eaa0-cc0b-4ee4-9b5a-2dfcd89a60b7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"9bc5a7bc-8f52-4db9-96d6-de5e68d52eaa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:25:35.629 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:35.645+0000 7fb2a648f640 10 librados: watch_flush enter 2024-03-29T11:25:35.629 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:35.645+0000 7fb2a648f640 10 librados: watch_flush exit 2024-03-29T11:25:35.629 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:35.645+0000 7fb2a648f640 20 client.4466.objecter shutdown clearing up homeless session... 2024-03-29T11:25:35.629 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:35.645+0000 7fb2a648f640 10 client.4466.objecter successfully canceled tick 2024-03-29T11:25:35.629 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:35.645+0000 7fb2a648f640 1 -- 172.21.15.84:0/1356351676 >> v1:172.21.15.179:6816/2981134980 conn(0x7fb28c06dc60 legacy=0x7fb28c070120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:35.629 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:35.645+0000 7fb2a648f640 10 monclient: shutdown 2024-03-29T11:25:35.630 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:35.645+0000 7fb2a648f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:25:35.630 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:35.645+0000 7fb2a648f640 1 -- 172.21.15.84:0/1356351676 >> v1:172.21.15.84:6790/0 conn(0x7fb2a01457f0 legacy=0x7fb2a01e5eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:35.630 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:35.645+0000 7fb2a648f640 1 -- 172.21.15.84:0/1356351676 shutdown_connections 2024-03-29T11:25:35.630 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:35.645+0000 7fb2a648f640 1 -- 172.21.15.84:0/1356351676 >> 172.21.15.84:0/1356351676 conn(0x7fb2a0066a00 msgr2=0x7fb2a00b4920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:25:35.630 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:35.646+0000 7fb2a648f640 1 -- 172.21.15.84:0/1356351676 shutdown_connections 2024-03-29T11:25:35.631 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:35.646+0000 7fb2a648f640 1 -- 172.21.15.84:0/1356351676 wait complete. 2024-03-29T11:25:35.631 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:35.646+0000 7fb2a648f640 1 librados: shutdown 2024-03-29T11:25:35.647 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:25:42.455 DEBUG:teuthology.orchestra.run.smithi084:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:25:42.607 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:42.621+0000 7fbc79104640 10 monclient: get_monmap_and_config 2024-03-29T11:25:42.607 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:42.621+0000 7fbc79104640 10 monclient: build_initial_monmap 2024-03-29T11:25:42.607 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:42.621+0000 7fbc79104640 10 monclient: monmap: 2024-03-29T11:25:42.607 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-29T11:25:42.607 INFO:teuthology.misc.health.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:25:42.607 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T11:25:42.622460+0000 2024-03-29T11:25:42.607 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T11:25:42.622460+0000 2024-03-29T11:25:42.607 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-29T11:25:42.607 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:25:42.607 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-29T11:25:42.608 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.noname-c 2024-03-29T11:25:42.608 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-b 2024-03-29T11:25:42.608 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:25:42.608 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:42.624+0000 7fbc79104640 1 Processor -- start 2024-03-29T11:25:42.609 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:42.625+0000 7fbc79104640 1 -- start start 2024-03-29T11:25:42.609 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:42.625+0000 7fbc79104640 10 monclient: init 2024-03-29T11:25:42.614 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:42.625+0000 7fbc79104640 10 monclient: _reopen_session rank -1 2024-03-29T11:25:42.614 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:42.626+0000 7fbc79104640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:25:42.614 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:42.626+0000 7fbc79104640 10 monclient(hunting): picked mon.noname-b con 0x7fbc740ae250 addr v1:172.21.15.84:6790/0 2024-03-29T11:25:42.614 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:42.626+0000 7fbc79104640 10 monclient(hunting): picked mon.noname-c con 0x7fbc7413f970 addr v1:172.21.15.179:6789/0 2024-03-29T11:25:42.614 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:42.626+0000 7fbc79104640 10 monclient(hunting): picked mon.noname-a con 0x7fbc74142800 addr v1:172.21.15.84:6789/0 2024-03-29T11:25:42.614 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:42.626+0000 7fbc79104640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbc740afe00 con 0x7fbc74142800 2024-03-29T11:25:42.614 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:42.626+0000 7fbc79104640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbc74142c10 con 0x7fbc7413f970 2024-03-29T11:25:42.614 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:42.626+0000 7fbc79104640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbc740afb20 con 0x7fbc740ae250 2024-03-29T11:25:42.614 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:42.626+0000 7fbc79104640 10 monclient(hunting): _renew_subs 2024-03-29T11:25:42.614 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:42.626+0000 7fbc79104640 10 monclient(hunting): authenticate will time out at 1203.920218s 2024-03-29T11:25:42.617 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:42.632+0000 7fbc7259c640 1 --1- >> v1:172.21.15.179:6789/0 conn(0x7fbc7413f970 0x7fbc741420f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.179:6789/0 says I am v1:172.21.15.84:54791/0 (socket says 172.21.15.84:54791) 2024-03-29T11:25:42.617 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:42.632+0000 7fbc7259c640 1 -- 172.21.15.84:0/2126738707 learned_addr learned my addr 172.21.15.84:0/2126738707 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-29T11:25:42.617 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:42.633+0000 7fbc71d9b640 1 -- 172.21.15.84:0/2126738707 <== mon.1 v1:172.21.15.179:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 171668887 0 0) 0x7fbc74142c10 con 0x7fbc7413f970 2024-03-29T11:25:42.617 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:42.633+0000 7fbc71d9b640 10 monclient(hunting): _negotiate 2024-03-29T11:25:42.617 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:42.633+0000 7fbc71d9b640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:25:42.617 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:42.633+0000 7fbc71d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:25:42.617 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:42.633+0000 7fbc71d9b640 10 monclient(hunting): my global_id is 4411 2024-03-29T11:25:42.617 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:42.633+0000 7fbc71d9b640 1 -- 172.21.15.84:0/2126738707 --> v1:172.21.15.179:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbc5c003520 con 0x7fbc7413f970 2024-03-29T11:25:42.618 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:42.633+0000 7fbc71d9b640 1 -- 172.21.15.84:0/2126738707 <== mon.1 v1:172.21.15.179:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3575673246 0 0) 0x7fbc5c003520 con 0x7fbc7413f970 2024-03-29T11:25:42.618 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:42.633+0000 7fbc71d9b640 1 -- 172.21.15.84:0/2126738707 >> v1:172.21.15.84:6790/0 conn(0x7fbc740ae250 legacy=0x7fbc7413f260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:42.618 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:42.633+0000 7fbc71d9b640 1 -- 172.21.15.84:0/2126738707 >> v1:172.21.15.84:6789/0 conn(0x7fbc74142800 legacy=0x7fbc7414da10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:42.618 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:42.633+0000 7fbc71d9b640 10 monclient: _finish_hunting 0 2024-03-29T11:25:42.618 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:42.633+0000 7fbc71d9b640 1 monclient: found mon.noname-c 2024-03-29T11:25:42.618 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:42.633+0000 7fbc71d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:25:42.618 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:42.633+0000 7fbc71d9b640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.179:6789/0 2024-03-29T11:25:42.622 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:42.633+0000 7fbc71d9b640 1 -- 172.21.15.84:0/2126738707 --> v1:172.21.15.179:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc740aab80 con 0x7fbc7413f970 2024-03-29T11:25:42.622 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:42.634+0000 7fbc71d9b640 10 monclient: _finish_auth 0 2024-03-29T11:25:42.622 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:42.634+0000 7fbc71d9b640 10 monclient: _check_auth_tickets 2024-03-29T11:25:42.622 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:42.634+0000 7fbc71d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:25:42.622 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:42.634+0000 7fbc71d9b640 1 -- 172.21.15.84:0/2126738707 <== mon.1 v1:172.21.15.179:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4236868694 0 0) 0x7fbc68001690 con 0x7fbc7413f970 2024-03-29T11:25:42.622 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:42.634+0000 7fbc71d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:25:42.622 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:42.634+0000 7fbc71d9b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:25:42.622 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:42.634+0000 7fbc71d9b640 10 monclient: dump: 2024-03-29T11:25:42.623 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T11:25:42.623 INFO:teuthology.misc.health.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:25:42.623 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T11:18:53.864025+0000 2024-03-29T11:25:42.623 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T11:18:53.864025+0000 2024-03-29T11:25:42.623 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T11:25:42.623 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:25:42.623 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T11:25:42.623 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T11:25:42.623 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T11:25:42.623 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:25:42.623 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:42.635+0000 7fbc79104640 5 monclient: authenticate success, global_id 4411 2024-03-29T11:25:42.623 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:42.635+0000 7fbc79104640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:25:42.623 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:42.637+0000 7fbc71d9b640 1 -- 172.21.15.84:0/2126738707 <== mon.1 v1:172.21.15.179:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbc68001870 con 0x7fbc7413f970 2024-03-29T11:25:42.623 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:42.637+0000 7fbc71d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:25:42.624 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:42.637+0000 7fbc79104640 10 monclient: get_monmap_and_config success 2024-03-29T11:25:42.624 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:42.637+0000 7fbc79104640 4 set_mon_vals no callback set 2024-03-29T11:25:42.624 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:42.638+0000 7fbc71d9b640 1 -- 172.21.15.84:0/2126738707 <== mon.1 v1:172.21.15.179:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4236868694 0 0) 0x7fbc68001c80 con 0x7fbc7413f970 2024-03-29T11:25:42.624 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:42.638+0000 7fbc71d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:25:42.624 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:42.638+0000 7fbc71d9b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:25:42.624 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:42.638+0000 7fbc71d9b640 10 monclient: dump: 2024-03-29T11:25:42.624 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T11:25:42.624 INFO:teuthology.misc.health.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:25:42.624 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T11:18:53.864025+0000 2024-03-29T11:25:42.624 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T11:18:53.864025+0000 2024-03-29T11:25:42.624 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T11:25:42.624 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:25:42.624 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T11:25:42.624 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T11:25:42.624 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T11:25:42.624 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:25:42.626 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:42.638+0000 7fbc79104640 10 monclient: shutdown 2024-03-29T11:25:42.626 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:42.638+0000 7fbc79104640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:25:42.626 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:42.638+0000 7fbc79104640 1 -- 172.21.15.84:0/2126738707 >> v1:172.21.15.179:6789/0 conn(0x7fbc7413f970 legacy=0x7fbc741420f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:42.626 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:42.638+0000 7fbc79104640 1 -- 172.21.15.84:0/2126738707 shutdown_connections 2024-03-29T11:25:42.626 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:42.638+0000 7fbc79104640 1 -- 172.21.15.84:0/2126738707 >> 172.21.15.84:0/2126738707 conn(0x7fbc740b35e0 msgr2=0x7fbc740b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:25:42.626 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:42.638+0000 7fbc79104640 1 -- 172.21.15.84:0/2126738707 shutdown_connections 2024-03-29T11:25:42.626 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:42.639+0000 7fbc79104640 1 -- 172.21.15.84:0/2126738707 wait complete. 2024-03-29T11:25:42.626 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:42.640+0000 7fbc79104640 10 monclient: build_initial_monmap 2024-03-29T11:25:42.626 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:42.640+0000 7fbc79104640 10 monclient: monmap: 2024-03-29T11:25:42.626 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-29T11:25:42.627 INFO:teuthology.misc.health.smithi084.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:25:42.627 INFO:teuthology.misc.health.smithi084.stderr:last_changed 0.000000 2024-03-29T11:25:42.627 INFO:teuthology.misc.health.smithi084.stderr:created 0.000000 2024-03-29T11:25:42.627 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-29T11:25:42.627 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:25:42.627 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-29T11:25:42.627 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.noname-b 2024-03-29T11:25:42.627 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-c 2024-03-29T11:25:42.627 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:25:42.627 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:42.640+0000 7fbc79104640 1 librados: starting msgr at 2024-03-29T11:25:42.627 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:42.640+0000 7fbc79104640 1 librados: starting objecter 2024-03-29T11:25:42.627 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:42.640+0000 7fbc79104640 1 Processor -- start 2024-03-29T11:25:42.630 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:42.641+0000 7fbc79104640 1 -- start start 2024-03-29T11:25:42.630 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:42.641+0000 7fbc79104640 1 librados: setting wanted keys 2024-03-29T11:25:42.630 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:42.641+0000 7fbc79104640 1 librados: calling monclient init 2024-03-29T11:25:42.630 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:42.641+0000 7fbc79104640 10 monclient: init 2024-03-29T11:25:42.630 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:42.642+0000 7fbc79104640 10 monclient: _reopen_session rank -1 2024-03-29T11:25:42.630 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:42.642+0000 7fbc79104640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:25:42.630 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:42.642+0000 7fbc79104640 10 monclient(hunting): picked mon.noname-a con 0x7fbc740ae250 addr v1:172.21.15.84:6789/0 2024-03-29T11:25:42.630 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:42.642+0000 7fbc79104640 10 monclient(hunting): picked mon.noname-b con 0x7fbc7413f970 addr v1:172.21.15.179:6789/0 2024-03-29T11:25:42.630 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:42.642+0000 7fbc79104640 10 monclient(hunting): picked mon.noname-c con 0x7fbc74142800 addr v1:172.21.15.84:6790/0 2024-03-29T11:25:42.630 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:42.642+0000 7fbc79104640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbc741450c0 con 0x7fbc740ae250 2024-03-29T11:25:42.630 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:42.642+0000 7fbc79104640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbc74144d10 con 0x7fbc7413f970 2024-03-29T11:25:42.630 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:42.642+0000 7fbc79104640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbc741434e0 con 0x7fbc74142800 2024-03-29T11:25:42.631 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:42.642+0000 7fbc79104640 10 monclient(hunting): _renew_subs 2024-03-29T11:25:42.631 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:42.642+0000 7fbc79104640 10 monclient(hunting): authenticate will time out at 1203.936302s 2024-03-29T11:25:42.631 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:42.645+0000 7fbc7259c640 1 --1- >> v1:172.21.15.179:6789/0 conn(0x7fbc7413f970 0x7fbc74071e60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.179:6789/0 says I am v1:172.21.15.84:54923/0 (socket says 172.21.15.84:54923) 2024-03-29T11:25:42.631 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:42.645+0000 7fbc7259c640 1 -- 172.21.15.84:0/2206790889 learned_addr learned my addr 172.21.15.84:0/2206790889 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-29T11:25:42.631 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:42.645+0000 7fbc5b7fe640 10 client.?.objecter ms_handle_connect 0x7fbc7413f970 2024-03-29T11:25:42.631 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:42.645+0000 7fbc5b7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:25:42.632 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:42.646+0000 7fbc5b7fe640 1 -- 172.21.15.84:0/2206790889 <== mon.1 v1:172.21.15.179:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3199554315 0 0) 0x7fbc74144d10 con 0x7fbc7413f970 2024-03-29T11:25:42.632 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:42.646+0000 7fbc5b7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:25:42.632 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:42.646+0000 7fbc5b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:25:42.632 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:42.646+0000 7fbc5b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:25:42.632 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:42.646+0000 7fbc5b7fe640 10 monclient(hunting): my global_id is 4414 2024-03-29T11:25:42.632 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:42.646+0000 7fbc5b7fe640 1 -- 172.21.15.84:0/2206790889 --> v1:172.21.15.179:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbc60004eb0 con 0x7fbc7413f970 2024-03-29T11:25:42.632 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:42.646+0000 7fbc5b7fe640 1 -- 172.21.15.84:0/2206790889 <== mon.1 v1:172.21.15.179:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1517821760 0 0) 0x7fbc60004eb0 con 0x7fbc7413f970 2024-03-29T11:25:42.632 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:42.646+0000 7fbc5b7fe640 1 -- 172.21.15.84:0/2206790889 --> v1:172.21.15.179:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbc74144d10 con 0x7fbc7413f970 2024-03-29T11:25:42.632 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:42.646+0000 7fbc5b7fe640 1 -- 172.21.15.84:0/2206790889 <== mon.1 v1:172.21.15.179:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4236868694 0 0) 0x7fbc68001b80 con 0x7fbc7413f970 2024-03-29T11:25:42.632 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:42.646+0000 7fbc5b7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:25:42.632 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:42.646+0000 7fbc5b7fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:25:42.632 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:42.646+0000 7fbc5b7fe640 10 monclient(hunting): dump: 2024-03-29T11:25:42.633 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T11:25:42.633 INFO:teuthology.misc.health.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:25:42.633 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T11:18:53.864025+0000 2024-03-29T11:25:42.633 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T11:18:53.864025+0000 2024-03-29T11:25:42.633 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T11:25:42.633 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:25:42.633 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T11:25:42.633 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T11:25:42.633 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T11:25:42.633 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:25:42.633 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:42.646+0000 7fbc5b7fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:25:42.633 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:42.646+0000 7fbc5b7fe640 1 -- 172.21.15.84:0/2206790889 <== mon.1 v1:172.21.15.179:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1860780613 0 0) 0x7fbc74144d10 con 0x7fbc7413f970 2024-03-29T11:25:42.633 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:42.646+0000 7fbc5b7fe640 1 -- 172.21.15.84:0/2206790889 >> v1:172.21.15.84:6790/0 conn(0x7fbc74142800 legacy=0x7fbc7413a940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:25:42.635 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:42.649+0000 7fbc5b7fe640 1 -- 172.21.15.84:0/2206790889 >> v1:172.21.15.84:6789/0 conn(0x7fbc740ae250 legacy=0x7fbc74144600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:42.635 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:42.649+0000 7fbc5b7fe640 10 monclient: _finish_hunting 0 2024-03-29T11:25:42.635 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:42.649+0000 7fbc5b7fe640 1 monclient: found mon.b 2024-03-29T11:25:42.635 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:42.649+0000 7fbc5b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:25:42.635 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:42.649+0000 7fbc5b7fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-29T11:25:42.635 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:42.649+0000 7fbc5b7fe640 1 -- 172.21.15.84:0/2206790889 --> v1:172.21.15.179:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc74072570 con 0x7fbc7413f970 2024-03-29T11:25:42.635 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:42.649+0000 7fbc5b7fe640 10 monclient: _finish_auth 0 2024-03-29T11:25:42.635 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:42.649+0000 7fbc5b7fe640 10 monclient: _check_auth_tickets 2024-03-29T11:25:42.635 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:42.649+0000 7fbc5b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:25:42.635 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:42.649+0000 7fbc79104640 5 monclient: authenticate success, global_id 4414 2024-03-29T11:25:42.635 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:42.649+0000 7fbc79104640 10 monclient: _renew_subs 2024-03-29T11:25:42.635 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:42.649+0000 7fbc79104640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-29T11:25:42.635 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:42.649+0000 7fbc79104640 1 -- 172.21.15.84:0/2206790889 --> v1:172.21.15.179:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbc74143ed0 con 0x7fbc7413f970 2024-03-29T11:25:42.636 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:42.649+0000 7fbc79104640 10 client.4414.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:25:42.636 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:42.649+0000 7fbc79104640 10 monclient: _renew_subs 2024-03-29T11:25:42.636 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:42.649+0000 7fbc79104640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-29T11:25:42.636 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:42.649+0000 7fbc79104640 1 -- 172.21.15.84:0/2206790889 --> v1:172.21.15.179:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fbc7413b050 con 0x7fbc7413f970 2024-03-29T11:25:42.636 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:42.649+0000 7fbc79104640 1 librados: init done 2024-03-29T11:25:42.636 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:42.652+0000 7fbc79104640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:25:42.637 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:42.652+0000 7fbc79104640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:25:42.637 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:42.652+0000 7fbc79104640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-29T11:25:42.637 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:42.652+0000 7fbc79104640 1 -- 172.21.15.84:0/2206790889 --> v1:172.21.15.179:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbc74144ea0 con 0x7fbc7413f970 2024-03-29T11:25:42.637 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:42.652+0000 7fbc5b7fe640 1 -- 172.21.15.84:0/2206790889 <== mon.1 v1:172.21.15.179:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbc68004bf0 con 0x7fbc7413f970 2024-03-29T11:25:42.637 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:42.652+0000 7fbc5b7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:25:42.637 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:42.652+0000 7fbc5b7fe640 1 -- 172.21.15.84:0/2206790889 <== mon.1 v1:172.21.15.179:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4236868694 0 0) 0x7fbc68004f00 con 0x7fbc7413f970 2024-03-29T11:25:42.637 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:42.652+0000 7fbc5b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:25:42.637 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:42.652+0000 7fbc5b7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:25:42.637 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:42.652+0000 7fbc5b7fe640 10 monclient: dump: 2024-03-29T11:25:42.637 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T11:25:42.637 INFO:teuthology.misc.health.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:25:42.637 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T11:18:53.864025+0000 2024-03-29T11:25:42.637 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T11:18:53.864025+0000 2024-03-29T11:25:42.637 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T11:25:42.638 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:25:42.638 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T11:25:42.638 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T11:25:42.638 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T11:25:42.638 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:25:42.639 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:42.652+0000 7fbc7159a640 4 set_mon_vals no callback set 2024-03-29T11:25:42.639 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:42.653+0000 7fbc5b7fe640 1 -- 172.21.15.84:0/2206790889 <== mon.1 v1:172.21.15.179:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4033942909 0 0) 0x7fbc6801aed0 con 0x7fbc7413f970 2024-03-29T11:25:42.639 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:42.654+0000 7fbc5b7fe640 1 -- 172.21.15.84:0/2206790889 <== mon.1 v1:172.21.15.179:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2808139587 0 0) 0x7fbc68084cb0 con 0x7fbc7413f970 2024-03-29T11:25:42.639 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:42.654+0000 7fbc5b7fe640 10 client.4414.objecter ms_dispatch 0x7fbc74000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:25:42.639 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:42.654+0000 7fbc5b7fe640 3 client.4414.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:25:42.639 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:42.654+0000 7fbc5b7fe640 3 client.4414.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:25:42.640 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:42.654+0000 7fbc5b7fe640 20 client.4414.objecter dump_active .. 0 homeless 2024-03-29T11:25:42.640 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:42.655+0000 7fbc72d9d640 10 monclient: get_auth_request con 0x7fbc6006dc00 auth_method 0 2024-03-29T11:25:42.640 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:42.655+0000 7fbc5b7fe640 10 client.4414.objecter ms_handle_connect 0x7fbc6006dc00 2024-03-29T11:25:42.644 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:42.660+0000 7fbc5b7fe640 1 -- 172.21.15.84:0/2206790889 <== mon.1 v1:172.21.15.179:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fbc68050590 con 0x7fbc7413f970 2024-03-29T11:25:42.644 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:42.660+0000 7fbc5b7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:25:42.644 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:42.660+0000 7fbc5b7fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:25:42.975 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:42.990+0000 7fbc79104640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:25:42.975 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:42.990+0000 7fbc79104640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:25:42.975 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:42.990+0000 7fbc79104640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-29T11:25:42.975 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:42.990+0000 7fbc79104640 1 -- 172.21.15.84:0/2206790889 --> v1:172.21.15.179:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fbc74075f10 con 0x7fbc7413f970 2024-03-29T11:25:42.978 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:42.993+0000 7fbc5b7fe640 1 -- 172.21.15.84:0/2206790889 <== mon.1 v1:172.21.15.179:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4209477549) 0x7fbc680524c0 con 0x7fbc7413f970 2024-03-29T11:25:42.978 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:42.993+0000 7fbc5b7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:25:42.978 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:42.993+0000 7fbc5b7fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:25:42.978 INFO:teuthology.misc.health.smithi084.stdout: 2024-03-29T11:25:42.979 INFO:teuthology.misc.health.smithi084.stdout:{"epoch":10,"fsid":"2c3e002f-9eb0-4860-853b-4ef1ef562213","created":"2024-03-29T11:19:32.029169+0000","modified":"2024-03-29T11:19:34.200299+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:34.200299+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"5c39e276-2034-4716-bbc0-57a634eefa92","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"cd580e99-109f-4bf1-88c0-a5987bc8b9ed","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"11e60a5a-051d-453f-a190-66387cb35592","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"2c329fbe-bbc4-4030-bf87-d711cad04679","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f4213d4a-bd36-4433-91ae-d7e2b02481d3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"f0cf44ea-6055-48af-805b-aed858102ff9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"d895eaa0-cc0b-4ee4-9b5a-2dfcd89a60b7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"9bc5a7bc-8f52-4db9-96d6-de5e68d52eaa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:25:42.981 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:42.997+0000 7fbc79104640 10 librados: watch_flush enter 2024-03-29T11:25:42.981 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:42.997+0000 7fbc79104640 10 librados: watch_flush exit 2024-03-29T11:25:42.982 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:42.997+0000 7fbc79104640 20 client.4414.objecter shutdown clearing up homeless session... 2024-03-29T11:25:42.982 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:42.997+0000 7fbc79104640 10 client.4414.objecter successfully canceled tick 2024-03-29T11:25:42.982 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:42.997+0000 7fbc79104640 1 -- 172.21.15.84:0/2206790889 >> v1:172.21.15.179:6816/2981134980 conn(0x7fbc6006dc00 legacy=0x7fbc600700c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:42.982 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:42.997+0000 7fbc79104640 10 monclient: shutdown 2024-03-29T11:25:42.982 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:42.997+0000 7fbc79104640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:25:42.983 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:42.997+0000 7fbc79104640 1 -- 172.21.15.84:0/2206790889 >> v1:172.21.15.179:6789/0 conn(0x7fbc7413f970 legacy=0x7fbc74071e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:42.983 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:42.998+0000 7fbc79104640 1 -- 172.21.15.84:0/2206790889 shutdown_connections 2024-03-29T11:25:42.983 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:42.998+0000 7fbc79104640 1 -- 172.21.15.84:0/2206790889 >> 172.21.15.84:0/2206790889 conn(0x7fbc740b35e0 msgr2=0x7fbc740b39c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:25:42.983 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:42.998+0000 7fbc79104640 1 -- 172.21.15.84:0/2206790889 shutdown_connections 2024-03-29T11:25:42.983 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:42.998+0000 7fbc79104640 1 -- 172.21.15.84:0/2206790889 wait complete. 2024-03-29T11:25:42.983 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:42.998+0000 7fbc79104640 1 librados: shutdown 2024-03-29T11:25:43.002 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:25:49.808 DEBUG:teuthology.orchestra.run.smithi084:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:25:49.954 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:49.969+0000 7f4926ee4640 10 monclient: get_monmap_and_config 2024-03-29T11:25:49.954 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:49.969+0000 7f4926ee4640 10 monclient: build_initial_monmap 2024-03-29T11:25:49.954 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:49.969+0000 7f4926ee4640 10 monclient: monmap: 2024-03-29T11:25:49.954 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-29T11:25:49.954 INFO:teuthology.misc.health.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:25:49.955 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T11:25:49.970553+0000 2024-03-29T11:25:49.955 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T11:25:49.970553+0000 2024-03-29T11:25:49.955 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-29T11:25:49.955 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:25:49.955 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-29T11:25:49.955 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.noname-c 2024-03-29T11:25:49.955 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-b 2024-03-29T11:25:49.955 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:25:49.975 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:49.970+0000 7f4926ee4640 1 Processor -- start 2024-03-29T11:25:49.975 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:49.970+0000 7f4926ee4640 1 -- start start 2024-03-29T11:25:49.975 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:49.970+0000 7f4926ee4640 10 monclient: init 2024-03-29T11:25:49.975 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:49.971+0000 7f4926ee4640 10 monclient: _reopen_session rank -1 2024-03-29T11:25:49.975 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:49.971+0000 7f4926ee4640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:25:49.975 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:49.971+0000 7f4926ee4640 10 monclient(hunting): picked mon.noname-c con 0x7f49201422b0 addr v1:172.21.15.179:6789/0 2024-03-29T11:25:49.975 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:49.971+0000 7f4926ee4640 10 monclient(hunting): picked mon.noname-a con 0x7f4920142690 addr v1:172.21.15.84:6789/0 2024-03-29T11:25:49.975 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:49.971+0000 7f4926ee4640 10 monclient(hunting): picked mon.noname-b con 0x7f49200b17d0 addr v1:172.21.15.84:6790/0 2024-03-29T11:25:49.976 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:49.971+0000 7f4926ee4640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4920142c80 con 0x7f4920142690 2024-03-29T11:25:49.976 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:49.971+0000 7f4926ee4640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f49200aae40 con 0x7f49201422b0 2024-03-29T11:25:49.976 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:49.971+0000 7f4926ee4640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f492013fbd0 con 0x7f49200b17d0 2024-03-29T11:25:49.976 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:49.971+0000 7f4926ee4640 10 monclient(hunting): _renew_subs 2024-03-29T11:25:49.976 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:49.971+0000 7f4926ee4640 10 monclient(hunting): authenticate will time out at 1211.265776s 2024-03-29T11:25:49.976 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:49.971+0000 7f4925481640 1 --1- >> v1:172.21.15.84:6790/0 conn(0x7f49200b17d0 0x7f49200b2ca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.84:6790/0 says I am v1:172.21.15.84:47308/0 (socket says 172.21.15.84:47308) 2024-03-29T11:25:49.976 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:49.971+0000 7f4925481640 1 -- 172.21.15.84:0/95309411 learned_addr learned my addr 172.21.15.84:0/95309411 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-29T11:25:49.976 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:49.972+0000 7f491f7fe640 1 -- 172.21.15.84:0/95309411 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3051724844 0 0) 0x7f492013fbd0 con 0x7f49200b17d0 2024-03-29T11:25:49.976 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:49.972+0000 7f491f7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:25:49.976 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:49.972+0000 7f491f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:25:49.976 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:49.972+0000 7f491f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:25:49.976 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:49.972+0000 7f491f7fe640 10 monclient(hunting): my global_id is 4472 2024-03-29T11:25:49.976 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:49.972+0000 7f491f7fe640 1 -- 172.21.15.84:0/95309411 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4910003520 con 0x7f49200b17d0 2024-03-29T11:25:49.977 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:49.972+0000 7f491f7fe640 1 -- 172.21.15.84:0/95309411 <== mon.1 v1:172.21.15.179:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2139924480 0 0) 0x7f49200aae40 con 0x7f49201422b0 2024-03-29T11:25:49.977 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:49.972+0000 7f491f7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:25:49.977 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:49.972+0000 7f491f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:25:49.977 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:49.972+0000 7f491f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:25:49.977 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:49.972+0000 7f491f7fe640 10 monclient(hunting): my global_id is 4417 2024-03-29T11:25:49.977 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:49.972+0000 7f491f7fe640 1 -- 172.21.15.84:0/95309411 --> v1:172.21.15.179:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f492013fbd0 con 0x7f49201422b0 2024-03-29T11:25:49.977 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:49.972+0000 7f491f7fe640 1 -- 172.21.15.84:0/95309411 <== mon.0 v1:172.21.15.84:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1109030053 0 0) 0x7f4920142c80 con 0x7f4920142690 2024-03-29T11:25:49.977 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:49.972+0000 7f491f7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:25:49.977 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:49.972+0000 7f491f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:25:49.977 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:49.972+0000 7f491f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:25:49.977 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:49.972+0000 7f491f7fe640 10 monclient(hunting): my global_id is 4455 2024-03-29T11:25:49.977 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:49.973+0000 7f491f7fe640 1 -- 172.21.15.84:0/95309411 --> v1:172.21.15.84:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f49200aae40 con 0x7f4920142690 2024-03-29T11:25:49.978 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:49.973+0000 7f491f7fe640 1 -- 172.21.15.84:0/95309411 <== mon.1 v1:172.21.15.179:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1774957678 0 0) 0x7f492013fbd0 con 0x7f49201422b0 2024-03-29T11:25:49.978 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:49.973+0000 7f491f7fe640 1 -- 172.21.15.84:0/95309411 >> v1:172.21.15.84:6790/0 conn(0x7f49200b17d0 legacy=0x7f49200b2ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:49.978 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:49.973+0000 7f491f7fe640 1 -- 172.21.15.84:0/95309411 >> v1:172.21.15.84:6789/0 conn(0x7f4920142690 legacy=0x7f49200b2590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:49.978 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:49.973+0000 7f491f7fe640 10 monclient: _finish_hunting 0 2024-03-29T11:25:49.978 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:49.973+0000 7f491f7fe640 1 monclient: found mon.noname-c 2024-03-29T11:25:49.978 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:49.973+0000 7f491f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:25:49.978 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:49.973+0000 7f491f7fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.179:6789/0 2024-03-29T11:25:49.978 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:49.973+0000 7f491f7fe640 1 -- 172.21.15.84:0/95309411 --> v1:172.21.15.179:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4920151940 con 0x7f49201422b0 2024-03-29T11:25:49.978 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:49.973+0000 7f491f7fe640 10 monclient: _finish_auth 0 2024-03-29T11:25:49.978 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:49.973+0000 7f491f7fe640 10 monclient: _check_auth_tickets 2024-03-29T11:25:49.978 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:49.973+0000 7f491f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:25:49.978 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:49.973+0000 7f491f7fe640 1 -- 172.21.15.84:0/95309411 <== mon.1 v1:172.21.15.179:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4236868694 0 0) 0x7f4914001670 con 0x7f49201422b0 2024-03-29T11:25:49.978 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:49.973+0000 7f491f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:25:49.979 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:49.973+0000 7f491f7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:25:49.979 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:49.973+0000 7f491f7fe640 10 monclient: dump: 2024-03-29T11:25:49.979 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T11:25:49.979 INFO:teuthology.misc.health.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:25:49.979 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T11:18:53.864025+0000 2024-03-29T11:25:49.979 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T11:18:53.864025+0000 2024-03-29T11:25:49.979 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T11:25:49.979 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:25:49.979 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T11:25:49.979 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T11:25:49.979 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T11:25:49.979 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:25:49.979 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:49.973+0000 7f4926ee4640 5 monclient: authenticate success, global_id 4417 2024-03-29T11:25:49.979 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:49.973+0000 7f4926ee4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:25:49.979 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:49.975+0000 7f491f7fe640 1 -- 172.21.15.84:0/95309411 <== mon.1 v1:172.21.15.179:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4914001890 con 0x7f49201422b0 2024-03-29T11:25:49.979 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:49.975+0000 7f491f7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:25:49.980 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:49.975+0000 7f4926ee4640 10 monclient: get_monmap_and_config success 2024-03-29T11:25:49.980 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:49.975+0000 7f4926ee4640 4 set_mon_vals no callback set 2024-03-29T11:25:49.980 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:49.975+0000 7f491f7fe640 1 -- 172.21.15.84:0/95309411 <== mon.1 v1:172.21.15.179:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4236868694 0 0) 0x7f4914001ba0 con 0x7f49201422b0 2024-03-29T11:25:49.980 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:49.975+0000 7f491f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:25:49.980 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:49.975+0000 7f491f7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:25:49.980 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:49.975+0000 7f491f7fe640 10 monclient: dump: 2024-03-29T11:25:49.980 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T11:25:49.980 INFO:teuthology.misc.health.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:25:49.980 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T11:18:53.864025+0000 2024-03-29T11:25:49.980 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T11:18:53.864025+0000 2024-03-29T11:25:49.980 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T11:25:49.980 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:25:49.980 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T11:25:49.980 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T11:25:49.980 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T11:25:49.981 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:25:49.981 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:49.976+0000 7f4926ee4640 10 monclient: shutdown 2024-03-29T11:25:49.981 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:49.976+0000 7f4926ee4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:25:49.981 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:49.976+0000 7f4926ee4640 1 -- 172.21.15.84:0/95309411 >> v1:172.21.15.179:6789/0 conn(0x7f49201422b0 legacy=0x7f492013f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:49.981 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:49.976+0000 7f4926ee4640 1 -- 172.21.15.84:0/95309411 shutdown_connections 2024-03-29T11:25:49.981 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:49.976+0000 7f4926ee4640 1 -- 172.21.15.84:0/95309411 >> 172.21.15.84:0/95309411 conn(0x7f4920066a00 msgr2=0x7f4920066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:25:49.981 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:49.976+0000 7f4926ee4640 1 -- 172.21.15.84:0/95309411 shutdown_connections 2024-03-29T11:25:49.981 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:49.976+0000 7f4926ee4640 1 -- 172.21.15.84:0/95309411 wait complete. 2024-03-29T11:25:49.981 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:49.977+0000 7f4926ee4640 10 monclient: build_initial_monmap 2024-03-29T11:25:49.981 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:49.977+0000 7f4926ee4640 10 monclient: monmap: 2024-03-29T11:25:49.981 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-29T11:25:49.981 INFO:teuthology.misc.health.smithi084.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:25:49.981 INFO:teuthology.misc.health.smithi084.stderr:last_changed 0.000000 2024-03-29T11:25:49.981 INFO:teuthology.misc.health.smithi084.stderr:created 0.000000 2024-03-29T11:25:49.981 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-29T11:25:49.982 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:25:49.982 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-29T11:25:49.982 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.noname-b 2024-03-29T11:25:49.982 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-c 2024-03-29T11:25:49.982 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:25:49.982 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:49.978+0000 7f4926ee4640 1 librados: starting msgr at 2024-03-29T11:25:49.982 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:49.978+0000 7f4926ee4640 1 librados: starting objecter 2024-03-29T11:25:49.982 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:49.978+0000 7f4926ee4640 1 Processor -- start 2024-03-29T11:25:49.982 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:49.978+0000 7f4926ee4640 1 -- start start 2024-03-29T11:25:49.982 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:49.978+0000 7f4926ee4640 1 librados: setting wanted keys 2024-03-29T11:25:49.982 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:49.978+0000 7f4926ee4640 1 librados: calling monclient init 2024-03-29T11:25:49.982 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:49.978+0000 7f4926ee4640 10 monclient: init 2024-03-29T11:25:49.982 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:49.979+0000 7f4926ee4640 10 monclient: _reopen_session rank -1 2024-03-29T11:25:49.983 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:49.979+0000 7f4926ee4640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:25:49.983 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:49.979+0000 7f4926ee4640 10 monclient(hunting): picked mon.noname-b con 0x7f49200b17d0 addr v1:172.21.15.179:6789/0 2024-03-29T11:25:49.983 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:49.979+0000 7f4926ee4640 10 monclient(hunting): picked mon.noname-c con 0x7f49201422b0 addr v1:172.21.15.84:6790/0 2024-03-29T11:25:49.983 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:49.979+0000 7f4926ee4640 10 monclient(hunting): picked mon.noname-a con 0x7f4920142690 addr v1:172.21.15.84:6789/0 2024-03-29T11:25:49.983 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:49.979+0000 7f4926ee4640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f49200b77b0 con 0x7f4920142690 2024-03-29T11:25:49.983 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:49.979+0000 7f4926ee4640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f49200ac030 con 0x7f49200b17d0 2024-03-29T11:25:49.983 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:49.979+0000 7f4926ee4640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f49200ac310 con 0x7f49201422b0 2024-03-29T11:25:49.983 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:49.979+0000 7f4926ee4640 10 monclient(hunting): _renew_subs 2024-03-29T11:25:49.983 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:49.979+0000 7f4926ee4640 10 monclient(hunting): authenticate will time out at 1211.273506s 2024-03-29T11:25:49.983 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:49.979+0000 7f4925481640 1 --1- >> v1:172.21.15.84:6789/0 conn(0x7f4920142690 0x7f492012ec20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.84:6789/0 says I am v1:172.21.15.84:50194/0 (socket says 172.21.15.84:50194) 2024-03-29T11:25:49.983 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:49.979+0000 7f4925481640 1 -- 172.21.15.84:0/580204222 learned_addr learned my addr 172.21.15.84:0/580204222 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-29T11:25:49.983 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:49.979+0000 7f491d7fa640 10 client.?.objecter ms_handle_connect 0x7f4920142690 2024-03-29T11:25:49.983 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:49.979+0000 7f491d7fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:25:49.983 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:49.980+0000 7f491d7fa640 10 client.?.objecter ms_handle_connect 0x7f49201422b0 2024-03-29T11:25:49.983 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:49.980+0000 7f491d7fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:25:49.984 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:49.980+0000 7f491d7fa640 1 -- 172.21.15.84:0/580204222 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 99084979 0 0) 0x7f49200ac310 con 0x7f49201422b0 2024-03-29T11:25:49.984 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:49.980+0000 7f491d7fa640 10 monclient(hunting): _negotiate 2024-03-29T11:25:49.984 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:49.980+0000 7f491d7fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:25:49.984 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:49.980+0000 7f491d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:25:49.984 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:49.980+0000 7f491d7fa640 10 monclient(hunting): my global_id is 4475 2024-03-29T11:25:49.984 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:49.980+0000 7f491d7fa640 1 -- 172.21.15.84:0/580204222 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4904004eb0 con 0x7f49201422b0 2024-03-29T11:25:49.984 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:49.980+0000 7f491d7fa640 10 client.?.objecter ms_handle_connect 0x7f49200b17d0 2024-03-29T11:25:49.984 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:49.980+0000 7f491d7fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:25:49.984 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:49.980+0000 7f491d7fa640 1 -- 172.21.15.84:0/580204222 <== mon.1 v1:172.21.15.179:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3474859933 0 0) 0x7f49200ac030 con 0x7f49200b17d0 2024-03-29T11:25:49.984 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:49.980+0000 7f491d7fa640 10 monclient(hunting): _negotiate 2024-03-29T11:25:49.984 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:49.980+0000 7f491d7fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:25:49.984 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:49.980+0000 7f491d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:25:49.984 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:49.980+0000 7f491d7fa640 10 monclient(hunting): my global_id is 4420 2024-03-29T11:25:49.984 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:49.980+0000 7f491d7fa640 1 -- 172.21.15.84:0/580204222 --> v1:172.21.15.179:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f49200ac310 con 0x7f49200b17d0 2024-03-29T11:25:49.984 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:49.980+0000 7f491d7fa640 1 -- 172.21.15.84:0/580204222 <== mon.0 v1:172.21.15.84:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1947729665 0 0) 0x7f49200b77b0 con 0x7f4920142690 2024-03-29T11:25:49.985 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:49.980+0000 7f491d7fa640 10 monclient(hunting): _negotiate 2024-03-29T11:25:49.985 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:49.980+0000 7f491d7fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:25:49.985 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:49.980+0000 7f491d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:25:49.985 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:49.980+0000 7f491d7fa640 10 monclient(hunting): my global_id is 4458 2024-03-29T11:25:49.985 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:49.980+0000 7f491d7fa640 1 -- 172.21.15.84:0/580204222 --> v1:172.21.15.84:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f49200ac030 con 0x7f4920142690 2024-03-29T11:25:49.985 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:49.981+0000 7f491d7fa640 1 -- 172.21.15.84:0/580204222 <== mon.2 v1:172.21.15.84:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2921215549 0 0) 0x7f4904004eb0 con 0x7f49201422b0 2024-03-29T11:25:49.985 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:49.981+0000 7f491d7fa640 1 -- 172.21.15.84:0/580204222 --> v1:172.21.15.84:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f49200b77b0 con 0x7f49201422b0 2024-03-29T11:25:49.985 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:49.981+0000 7f491d7fa640 1 -- 172.21.15.84:0/580204222 <== mon.2 v1:172.21.15.84:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4236868694 0 0) 0x7f490c001850 con 0x7f49201422b0 2024-03-29T11:25:49.985 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:49.981+0000 7f491d7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:25:49.985 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:49.981+0000 7f491d7fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:25:49.985 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:49.981+0000 7f491d7fa640 10 monclient(hunting): dump: 2024-03-29T11:25:49.986 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T11:25:49.986 INFO:teuthology.misc.health.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:25:49.986 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T11:18:53.864025+0000 2024-03-29T11:25:49.986 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T11:18:53.864025+0000 2024-03-29T11:25:49.986 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T11:25:49.986 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:25:49.986 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T11:25:49.986 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T11:25:49.986 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T11:25:49.987 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:25:49.987 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:49.981+0000 7f491d7fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:25:49.987 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:49.981+0000 7f491d7fa640 1 -- 172.21.15.84:0/580204222 <== mon.1 v1:172.21.15.179:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2492956984 0 0) 0x7f49200ac310 con 0x7f49200b17d0 2024-03-29T11:25:49.987 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:49.981+0000 7f491d7fa640 1 -- 172.21.15.84:0/580204222 --> v1:172.21.15.179:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4904004eb0 con 0x7f49200b17d0 2024-03-29T11:25:49.987 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:49.981+0000 7f491d7fa640 1 -- 172.21.15.84:0/580204222 <== mon.1 v1:172.21.15.179:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4236868694 0 0) 0x7f4914001530 con 0x7f49200b17d0 2024-03-29T11:25:49.988 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:49.981+0000 7f491d7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:25:49.988 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:49.981+0000 7f491d7fa640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:25:49.988 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:49.981+0000 7f491d7fa640 10 monclient(hunting): dump: 2024-03-29T11:25:49.988 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T11:25:49.988 INFO:teuthology.misc.health.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:25:49.988 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T11:18:53.864025+0000 2024-03-29T11:25:49.988 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T11:18:53.864025+0000 2024-03-29T11:25:49.988 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T11:25:49.988 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:25:49.988 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T11:25:49.988 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T11:25:49.989 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T11:25:49.989 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:25:49.989 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:49.981+0000 7f491d7fa640 1 -- 172.21.15.84:0/580204222 <== mon.0 v1:172.21.15.84:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2333865110 0 0) 0x7f49200ac030 con 0x7f4920142690 2024-03-29T11:25:49.989 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:49.981+0000 7f491d7fa640 1 -- 172.21.15.84:0/580204222 --> v1:172.21.15.84:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f49200ac310 con 0x7f4920142690 2024-03-29T11:25:49.989 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:49.981+0000 7f491d7fa640 1 -- 172.21.15.84:0/580204222 <== mon.0 v1:172.21.15.84:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4236868694 0 0) 0x7f4918001490 con 0x7f4920142690 2024-03-29T11:25:49.989 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:49.981+0000 7f491d7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:25:49.989 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:49.981+0000 7f491d7fa640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:25:49.989 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:49.982+0000 7f491d7fa640 10 monclient(hunting): dump: 2024-03-29T11:25:49.989 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T11:25:49.989 INFO:teuthology.misc.health.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:25:49.989 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T11:18:53.864025+0000 2024-03-29T11:25:49.989 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T11:18:53.864025+0000 2024-03-29T11:25:49.989 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T11:25:49.989 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:25:49.990 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T11:25:49.990 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T11:25:49.990 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T11:25:49.990 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:25:49.990 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:49.982+0000 7f491d7fa640 1 -- 172.21.15.84:0/580204222 <== mon.0 v1:172.21.15.84:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 163513410 0 0) 0x7f49200ac310 con 0x7f4920142690 2024-03-29T11:25:49.990 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:49.982+0000 7f491d7fa640 1 -- 172.21.15.84:0/580204222 >> v1:172.21.15.84:6790/0 conn(0x7f49201422b0 legacy=0x7f49201300b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:49.990 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:49.982+0000 7f491d7fa640 1 -- 172.21.15.84:0/580204222 >> v1:172.21.15.179:6789/0 conn(0x7f49200b17d0 legacy=0x7f49200b6aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:49.990 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:49.982+0000 7f491d7fa640 10 monclient: _finish_hunting 0 2024-03-29T11:25:49.990 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:49.982+0000 7f491d7fa640 1 monclient: found mon.a 2024-03-29T11:25:49.990 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:49.982+0000 7f491d7fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:25:49.990 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:49.982+0000 7f491d7fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.84:6789/0 2024-03-29T11:25:49.991 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:49.982+0000 7f491d7fa640 1 -- 172.21.15.84:0/580204222 --> v1:172.21.15.84:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49200b7290 con 0x7f4920142690 2024-03-29T11:25:49.991 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:49.982+0000 7f491d7fa640 10 monclient: _finish_auth 0 2024-03-29T11:25:49.991 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:49.982+0000 7f491d7fa640 10 monclient: _check_auth_tickets 2024-03-29T11:25:49.991 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:49.982+0000 7f491d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:25:49.991 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:49.982+0000 7f4926ee4640 5 monclient: authenticate success, global_id 4458 2024-03-29T11:25:49.991 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:49.982+0000 7f4926ee4640 10 monclient: _renew_subs 2024-03-29T11:25:49.991 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:49.982+0000 7f4926ee4640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.84:6789/0 2024-03-29T11:25:49.992 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:49.982+0000 7f4926ee4640 1 -- 172.21.15.84:0/580204222 --> v1:172.21.15.84:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f49201307c0 con 0x7f4920142690 2024-03-29T11:25:49.992 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:49.982+0000 7f4926ee4640 10 client.4458.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:25:49.992 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:49.982+0000 7f4926ee4640 10 monclient: _renew_subs 2024-03-29T11:25:49.992 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:49.982+0000 7f4926ee4640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.84:6789/0 2024-03-29T11:25:49.992 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:49.982+0000 7f4926ee4640 1 -- 172.21.15.84:0/580204222 --> v1:172.21.15.84:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f49200b6510 con 0x7f4920142690 2024-03-29T11:25:49.992 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:49.982+0000 7f4926ee4640 1 librados: init done 2024-03-29T11:25:49.992 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:49.983+0000 7f491d7fa640 1 -- 172.21.15.84:0/580204222 <== mon.0 v1:172.21.15.84:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4918002000 con 0x7f4920142690 2024-03-29T11:25:49.992 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:49.983+0000 7f491d7fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:25:49.992 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:49.983+0000 7f491d7fa640 1 -- 172.21.15.84:0/580204222 <== mon.0 v1:172.21.15.84:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4236868694 0 0) 0x7f4918004d00 con 0x7f4920142690 2024-03-29T11:25:49.992 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:49.983+0000 7f491d7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:25:49.992 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:49.983+0000 7f491d7fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:25:49.993 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:49.983+0000 7f491effd640 4 set_mon_vals no callback set 2024-03-29T11:25:49.993 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:49.983+0000 7f491d7fa640 10 monclient: dump: 2024-03-29T11:25:49.993 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T11:25:49.993 INFO:teuthology.misc.health.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:25:49.993 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T11:18:53.864025+0000 2024-03-29T11:25:49.993 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T11:18:53.864025+0000 2024-03-29T11:25:49.993 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T11:25:49.993 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:25:49.993 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T11:25:49.993 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T11:25:49.993 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T11:25:49.993 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:25:49.993 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:49.985+0000 7f4926ee4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:25:49.994 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:49.985+0000 7f4926ee4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:25:49.994 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:49.985+0000 7f4926ee4640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.84:6789/0 2024-03-29T11:25:49.994 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:49.985+0000 7f4926ee4640 1 -- 172.21.15.84:0/580204222 --> v1:172.21.15.84:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4900005080 con 0x7f4920142690 2024-03-29T11:25:49.994 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:49.986+0000 7f491d7fa640 1 -- 172.21.15.84:0/580204222 <== mon.0 v1:172.21.15.84:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4033942909 0 0) 0x7f491801acd0 con 0x7f4920142690 2024-03-29T11:25:49.994 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:49.986+0000 7f491d7fa640 1 -- 172.21.15.84:0/580204222 <== mon.0 v1:172.21.15.84:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2808139587 0 0) 0x7f4918084c20 con 0x7f4920142690 2024-03-29T11:25:49.994 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:49.987+0000 7f491d7fa640 10 client.4458.objecter ms_dispatch 0x7f4920000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:25:49.994 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:49.987+0000 7f491d7fa640 3 client.4458.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:25:49.994 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:49.987+0000 7f4924c80640 10 monclient: get_auth_request con 0x7f490406e0a0 auth_method 0 2024-03-29T11:25:49.994 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:49.987+0000 7f491d7fa640 3 client.4458.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:25:49.994 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:49.987+0000 7f491d7fa640 20 client.4458.objecter dump_active .. 0 homeless 2024-03-29T11:25:49.994 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:49.987+0000 7f491d7fa640 10 client.4458.objecter ms_handle_connect 0x7f490406e0a0 2024-03-29T11:25:49.995 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:49.994+0000 7f491d7fa640 1 -- 172.21.15.84:0/580204222 <== mon.0 v1:172.21.15.84:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f4918050330 con 0x7f4920142690 2024-03-29T11:25:49.995 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:49.994+0000 7f491d7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:25:49.995 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:49.994+0000 7f491d7fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:25:50.323 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:50.338+0000 7f4926ee4640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:25:50.323 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:50.338+0000 7f4926ee4640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:25:50.323 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:50.338+0000 7f4926ee4640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.84:6789/0 2024-03-29T11:25:50.323 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:50.338+0000 7f4926ee4640 1 -- 172.21.15.84:0/580204222 --> v1:172.21.15.84:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4900005330 con 0x7f4920142690 2024-03-29T11:25:50.324 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:50.340+0000 7f491d7fa640 1 -- 172.21.15.84:0/580204222 <== mon.0 v1:172.21.15.84:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4209477549) 0x7f4918052430 con 0x7f4920142690 2024-03-29T11:25:50.324 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:50.340+0000 7f491d7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:25:50.325 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:50.340+0000 7f491d7fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:25:50.325 INFO:teuthology.misc.health.smithi084.stdout: 2024-03-29T11:25:50.325 INFO:teuthology.misc.health.smithi084.stdout:{"epoch":10,"fsid":"2c3e002f-9eb0-4860-853b-4ef1ef562213","created":"2024-03-29T11:19:32.029169+0000","modified":"2024-03-29T11:19:34.200299+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:34.200299+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"5c39e276-2034-4716-bbc0-57a634eefa92","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"cd580e99-109f-4bf1-88c0-a5987bc8b9ed","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"11e60a5a-051d-453f-a190-66387cb35592","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"2c329fbe-bbc4-4030-bf87-d711cad04679","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f4213d4a-bd36-4433-91ae-d7e2b02481d3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"f0cf44ea-6055-48af-805b-aed858102ff9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"d895eaa0-cc0b-4ee4-9b5a-2dfcd89a60b7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"9bc5a7bc-8f52-4db9-96d6-de5e68d52eaa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:25:50.329 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:50.344+0000 7f4926ee4640 10 librados: watch_flush enter 2024-03-29T11:25:50.329 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:50.344+0000 7f4926ee4640 10 librados: watch_flush exit 2024-03-29T11:25:50.329 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:50.344+0000 7f4926ee4640 20 client.4458.objecter shutdown clearing up homeless session... 2024-03-29T11:25:50.329 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:50.344+0000 7f4926ee4640 10 client.4458.objecter successfully canceled tick 2024-03-29T11:25:50.329 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:50.344+0000 7f4926ee4640 1 -- 172.21.15.84:0/580204222 >> v1:172.21.15.179:6816/2981134980 conn(0x7f490406e0a0 legacy=0x7f4904070560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:50.329 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:50.344+0000 7f4926ee4640 10 monclient: shutdown 2024-03-29T11:25:50.329 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:50.344+0000 7f4926ee4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:25:50.329 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:50.344+0000 7f4926ee4640 1 -- 172.21.15.84:0/580204222 >> v1:172.21.15.84:6789/0 conn(0x7f4920142690 legacy=0x7f492012ec20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:50.329 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:50.344+0000 7f4926ee4640 1 -- 172.21.15.84:0/580204222 shutdown_connections 2024-03-29T11:25:50.329 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:50.344+0000 7f4926ee4640 1 -- 172.21.15.84:0/580204222 >> 172.21.15.84:0/580204222 conn(0x7f4920066a00 msgr2=0x7f492013bdf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:25:50.329 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:50.344+0000 7f4926ee4640 1 -- 172.21.15.84:0/580204222 shutdown_connections 2024-03-29T11:25:50.329 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:50.344+0000 7f4926ee4640 1 -- 172.21.15.84:0/580204222 wait complete. 2024-03-29T11:25:50.330 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:50.344+0000 7f4926ee4640 1 librados: shutdown 2024-03-29T11:25:50.346 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:25:57.153 DEBUG:teuthology.orchestra.run.smithi084:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:25:57.304 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:57.318+0000 7fb7cf688640 10 monclient: get_monmap_and_config 2024-03-29T11:25:57.304 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:57.318+0000 7fb7cf688640 10 monclient: build_initial_monmap 2024-03-29T11:25:57.304 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:57.319+0000 7fb7cf688640 10 monclient: monmap: 2024-03-29T11:25:57.304 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-29T11:25:57.304 INFO:teuthology.misc.health.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:25:57.304 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T11:25:57.319919+0000 2024-03-29T11:25:57.304 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T11:25:57.319919+0000 2024-03-29T11:25:57.304 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-29T11:25:57.304 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:25:57.305 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-29T11:25:57.305 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.noname-c 2024-03-29T11:25:57.305 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-b 2024-03-29T11:25:57.305 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:25:57.305 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:57.319+0000 7fb7cf688640 1 Processor -- start 2024-03-29T11:25:57.305 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:57.320+0000 7fb7cf688640 1 -- start start 2024-03-29T11:25:57.305 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:57.320+0000 7fb7cf688640 10 monclient: init 2024-03-29T11:25:57.307 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:57.320+0000 7fb7cf688640 10 monclient: _reopen_session rank -1 2024-03-29T11:25:57.307 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:57.320+0000 7fb7cf688640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:25:57.307 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:57.321+0000 7fb7cf688640 10 monclient(hunting): picked mon.noname-c con 0x7fb7c80b0500 addr v1:172.21.15.179:6789/0 2024-03-29T11:25:57.308 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:57.321+0000 7fb7cf688640 10 monclient(hunting): picked mon.noname-b con 0x7fb7c80b08e0 addr v1:172.21.15.84:6790/0 2024-03-29T11:25:57.308 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:57.321+0000 7fb7cf688640 10 monclient(hunting): picked mon.noname-a con 0x7fb7c80aeca0 addr v1:172.21.15.84:6789/0 2024-03-29T11:25:57.308 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:57.321+0000 7fb7cf688640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb7c8140620 con 0x7fb7c80aeca0 2024-03-29T11:25:57.308 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:57.321+0000 7fb7cf688640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb7c813daf0 con 0x7fb7c80b0500 2024-03-29T11:25:57.308 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:57.321+0000 7fb7cf688640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb7c80ae180 con 0x7fb7c80b08e0 2024-03-29T11:25:57.308 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:57.321+0000 7fb7cf688640 10 monclient(hunting): _renew_subs 2024-03-29T11:25:57.308 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:57.321+0000 7fb7cf688640 10 monclient(hunting): authenticate will time out at 1218.615202s 2024-03-29T11:25:57.308 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:57.321+0000 7fb7cd424640 1 --1- >> v1:172.21.15.179:6789/0 conn(0x7fb7c80b0500 0x7fb7c813ff10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.179:6789/0 says I am v1:172.21.15.84:52180/0 (socket says 172.21.15.84:52180) 2024-03-29T11:25:57.308 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:57.321+0000 7fb7cd424640 1 -- 172.21.15.84:0/1855058375 learned_addr learned my addr 172.21.15.84:0/1855058375 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-29T11:25:57.308 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:57.321+0000 7fb7c7fff640 1 -- 172.21.15.84:0/1855058375 <== mon.1 v1:172.21.15.179:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2591962452 0 0) 0x7fb7c813daf0 con 0x7fb7c80b0500 2024-03-29T11:25:57.308 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:57.321+0000 7fb7c7fff640 10 monclient(hunting): _negotiate 2024-03-29T11:25:57.308 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:57.321+0000 7fb7c7fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:25:57.308 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:57.321+0000 7fb7c7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:25:57.308 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:57.321+0000 7fb7c7fff640 10 monclient(hunting): my global_id is 4423 2024-03-29T11:25:57.308 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:57.321+0000 7fb7c7fff640 1 -- 172.21.15.84:0/1855058375 --> v1:172.21.15.179:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb7b0003520 con 0x7fb7c80b0500 2024-03-29T11:25:57.309 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:57.321+0000 7fb7c7fff640 1 -- 172.21.15.84:0/1855058375 <== mon.0 v1:172.21.15.84:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3220855 0 0) 0x7fb7c8140620 con 0x7fb7c80aeca0 2024-03-29T11:25:57.309 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:57.321+0000 7fb7c7fff640 10 monclient(hunting): _negotiate 2024-03-29T11:25:57.309 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:57.321+0000 7fb7c7fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:25:57.309 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:57.321+0000 7fb7c7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:25:57.309 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:57.321+0000 7fb7c7fff640 10 monclient(hunting): my global_id is 4461 2024-03-29T11:25:57.309 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:57.321+0000 7fb7c7fff640 1 -- 172.21.15.84:0/1855058375 --> v1:172.21.15.84:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb7c813daf0 con 0x7fb7c80aeca0 2024-03-29T11:25:57.309 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:57.322+0000 7fb7c7fff640 1 -- 172.21.15.84:0/1855058375 <== mon.0 v1:172.21.15.84:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1792343784 0 0) 0x7fb7c813daf0 con 0x7fb7c80aeca0 2024-03-29T11:25:57.309 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:57.322+0000 7fb7c7fff640 1 -- 172.21.15.84:0/1855058375 >> v1:172.21.15.84:6790/0 conn(0x7fb7c80b08e0 legacy=0x7fb7c813d3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:25:57.309 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:57.322+0000 7fb7c7fff640 1 -- 172.21.15.84:0/1855058375 >> v1:172.21.15.179:6789/0 conn(0x7fb7c80b0500 legacy=0x7fb7c813ff10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:57.309 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:57.322+0000 7fb7c7fff640 10 monclient: _finish_hunting 0 2024-03-29T11:25:57.309 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:57.322+0000 7fb7c7fff640 1 monclient: found mon.noname-a 2024-03-29T11:25:57.309 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:57.322+0000 7fb7c7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:25:57.309 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:57.322+0000 7fb7c7fff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.84:6789/0 2024-03-29T11:25:57.310 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:57.322+0000 7fb7c7fff640 1 -- 172.21.15.84:0/1855058375 --> v1:172.21.15.84:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7c80accf0 con 0x7fb7c80aeca0 2024-03-29T11:25:57.310 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:57.322+0000 7fb7c7fff640 10 monclient: _finish_auth 0 2024-03-29T11:25:57.310 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:57.322+0000 7fb7c7fff640 10 monclient: _check_auth_tickets 2024-03-29T11:25:57.310 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:57.322+0000 7fb7c7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:25:57.310 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:57.322+0000 7fb7cf688640 5 monclient: authenticate success, global_id 4461 2024-03-29T11:25:57.310 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:57.322+0000 7fb7cf688640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:25:57.310 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:57.322+0000 7fb7c7fff640 1 -- 172.21.15.84:0/1855058375 <== mon.0 v1:172.21.15.84:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4236868694 0 0) 0x7fb7c00019f0 con 0x7fb7c80aeca0 2024-03-29T11:25:57.310 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:57.322+0000 7fb7c7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:25:57.310 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:57.322+0000 7fb7c7fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:25:57.310 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:57.322+0000 7fb7c7fff640 10 monclient: dump: 2024-03-29T11:25:57.310 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T11:25:57.310 INFO:teuthology.misc.health.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:25:57.310 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T11:18:53.864025+0000 2024-03-29T11:25:57.310 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T11:18:53.864025+0000 2024-03-29T11:25:57.311 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T11:25:57.311 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:25:57.311 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T11:25:57.311 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T11:25:57.311 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T11:25:57.311 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:25:57.311 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:57.322+0000 7fb7cf688640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:25:57.311 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:57.322+0000 7fb7c7fff640 1 -- 172.21.15.84:0/1855058375 <== mon.0 v1:172.21.15.84:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb7c0001bc0 con 0x7fb7c80aeca0 2024-03-29T11:25:57.311 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:57.322+0000 7fb7c7fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:25:57.311 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:57.322+0000 7fb7cf688640 10 monclient: get_monmap_and_config success 2024-03-29T11:25:57.311 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:57.322+0000 7fb7cf688640 4 set_mon_vals no callback set 2024-03-29T11:25:57.311 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:57.322+0000 7fb7c7fff640 1 -- 172.21.15.84:0/1855058375 <== mon.0 v1:172.21.15.84:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4236868694 0 0) 0x7fb7c0001f50 con 0x7fb7c80aeca0 2024-03-29T11:25:57.311 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:57.322+0000 7fb7c7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:25:57.311 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:57.322+0000 7fb7c7fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:25:57.311 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:57.322+0000 7fb7c7fff640 10 monclient: dump: 2024-03-29T11:25:57.312 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T11:25:57.312 INFO:teuthology.misc.health.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:25:57.312 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T11:18:53.864025+0000 2024-03-29T11:25:57.312 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T11:18:53.864025+0000 2024-03-29T11:25:57.312 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T11:25:57.312 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:25:57.312 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T11:25:57.312 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T11:25:57.312 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T11:25:57.312 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:25:57.318 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:57.323+0000 7fb7cf688640 10 monclient: shutdown 2024-03-29T11:25:57.318 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:57.323+0000 7fb7cf688640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:25:57.318 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:57.323+0000 7fb7cf688640 1 -- 172.21.15.84:0/1855058375 >> v1:172.21.15.84:6789/0 conn(0x7fb7c80aeca0 legacy=0x7fb7c814b7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:57.318 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:57.323+0000 7fb7cf688640 1 -- 172.21.15.84:0/1855058375 shutdown_connections 2024-03-29T11:25:57.318 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:57.323+0000 7fb7cf688640 1 -- 172.21.15.84:0/1855058375 >> 172.21.15.84:0/1855058375 conn(0x7fb7c806b660 msgr2=0x7fb7c806ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:25:57.318 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:57.323+0000 7fb7cf688640 1 -- 172.21.15.84:0/1855058375 shutdown_connections 2024-03-29T11:25:57.318 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:57.323+0000 7fb7cf688640 1 -- 172.21.15.84:0/1855058375 wait complete. 2024-03-29T11:25:57.318 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:57.324+0000 7fb7cf688640 10 monclient: build_initial_monmap 2024-03-29T11:25:57.319 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:57.324+0000 7fb7cf688640 10 monclient: monmap: 2024-03-29T11:25:57.319 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-29T11:25:57.319 INFO:teuthology.misc.health.smithi084.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:25:57.319 INFO:teuthology.misc.health.smithi084.stderr:last_changed 0.000000 2024-03-29T11:25:57.319 INFO:teuthology.misc.health.smithi084.stderr:created 0.000000 2024-03-29T11:25:57.319 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-29T11:25:57.319 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:25:57.319 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-29T11:25:57.319 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.noname-b 2024-03-29T11:25:57.319 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-c 2024-03-29T11:25:57.319 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:25:57.319 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:57.325+0000 7fb7cf688640 1 librados: starting msgr at 2024-03-29T11:25:57.319 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:57.325+0000 7fb7cf688640 1 librados: starting objecter 2024-03-29T11:25:57.319 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:57.325+0000 7fb7cf688640 1 Processor -- start 2024-03-29T11:25:57.319 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:57.325+0000 7fb7cf688640 1 -- start start 2024-03-29T11:25:57.320 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:57.325+0000 7fb7cf688640 1 librados: setting wanted keys 2024-03-29T11:25:57.320 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:57.325+0000 7fb7cf688640 1 librados: calling monclient init 2024-03-29T11:25:57.320 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:57.325+0000 7fb7cf688640 10 monclient: init 2024-03-29T11:25:57.320 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:57.326+0000 7fb7cf688640 10 monclient: _reopen_session rank -1 2024-03-29T11:25:57.320 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:57.326+0000 7fb7cf688640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:25:57.320 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:57.326+0000 7fb7cf688640 10 monclient(hunting): picked mon.noname-a con 0x7fb7c80aeca0 addr v1:172.21.15.84:6789/0 2024-03-29T11:25:57.320 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:57.326+0000 7fb7cf688640 10 monclient(hunting): picked mon.noname-c con 0x7fb7c80b0500 addr v1:172.21.15.84:6790/0 2024-03-29T11:25:57.320 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:57.326+0000 7fb7cf688640 10 monclient(hunting): picked mon.noname-b con 0x7fb7c806dd90 addr v1:172.21.15.179:6789/0 2024-03-29T11:25:57.320 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:57.326+0000 7fb7cf688640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb7c806fc30 con 0x7fb7c80aeca0 2024-03-29T11:25:57.320 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:57.326+0000 7fb7cf688640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb7c806fea0 con 0x7fb7c806dd90 2024-03-29T11:25:57.320 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:57.326+0000 7fb7cf688640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb7c8070180 con 0x7fb7c80b0500 2024-03-29T11:25:57.320 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:57.326+0000 7fb7cf688640 10 monclient(hunting): _renew_subs 2024-03-29T11:25:57.320 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:57.326+0000 7fb7cf688640 10 monclient(hunting): authenticate will time out at 1218.620528s 2024-03-29T11:25:57.320 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:57.326+0000 7fb7cd424640 1 --1- >> v1:172.21.15.84:6789/0 conn(0x7fb7c80aeca0 0x7fb7c806e6e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.84:6789/0 says I am v1:172.21.15.84:57268/0 (socket says 172.21.15.84:57268) 2024-03-29T11:25:57.320 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:57.326+0000 7fb7cd424640 1 -- 172.21.15.84:0/3862013000 learned_addr learned my addr 172.21.15.84:0/3862013000 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-29T11:25:57.321 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:57.326+0000 7fb7c5ffb640 10 client.?.objecter ms_handle_connect 0x7fb7c80aeca0 2024-03-29T11:25:57.321 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:57.326+0000 7fb7c5ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:25:57.321 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:57.326+0000 7fb7c5ffb640 10 client.?.objecter ms_handle_connect 0x7fb7c806dd90 2024-03-29T11:25:57.321 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:57.326+0000 7fb7c5ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:25:57.321 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:57.326+0000 7fb7c5ffb640 10 client.?.objecter ms_handle_connect 0x7fb7c80b0500 2024-03-29T11:25:57.321 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:57.326+0000 7fb7c5ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:25:57.321 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:57.327+0000 7fb7c5ffb640 1 -- 172.21.15.84:0/3862013000 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1836530779 0 0) 0x7fb7c8070180 con 0x7fb7c80b0500 2024-03-29T11:25:57.321 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:57.327+0000 7fb7c5ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:25:57.321 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:57.327+0000 7fb7c5ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:25:57.321 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:57.327+0000 7fb7c5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:25:57.321 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:57.327+0000 7fb7c5ffb640 10 monclient(hunting): my global_id is 4478 2024-03-29T11:25:57.321 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:57.327+0000 7fb7c5ffb640 1 -- 172.21.15.84:0/3862013000 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb7b4004eb0 con 0x7fb7c80b0500 2024-03-29T11:25:57.321 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:57.327+0000 7fb7c5ffb640 1 -- 172.21.15.84:0/3862013000 <== mon.2 v1:172.21.15.84:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1831205176 0 0) 0x7fb7b4004eb0 con 0x7fb7c80b0500 2024-03-29T11:25:57.321 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:57.327+0000 7fb7c5ffb640 1 -- 172.21.15.84:0/3862013000 --> v1:172.21.15.84:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb7c8070180 con 0x7fb7c80b0500 2024-03-29T11:25:57.322 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:57.327+0000 7fb7c5ffb640 1 -- 172.21.15.84:0/3862013000 <== mon.2 v1:172.21.15.84:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4236868694 0 0) 0x7fb7b8001700 con 0x7fb7c80b0500 2024-03-29T11:25:57.322 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:57.327+0000 7fb7c5ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:25:57.322 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:57.327+0000 7fb7c5ffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:25:57.322 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:57.327+0000 7fb7c5ffb640 10 monclient(hunting): dump: 2024-03-29T11:25:57.322 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T11:25:57.322 INFO:teuthology.misc.health.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:25:57.322 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T11:18:53.864025+0000 2024-03-29T11:25:57.322 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T11:18:53.864025+0000 2024-03-29T11:25:57.322 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T11:25:57.322 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:25:57.322 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T11:25:57.322 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T11:25:57.322 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T11:25:57.322 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:25:57.322 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:57.327+0000 7fb7c5ffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:25:57.323 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:57.327+0000 7fb7c5ffb640 1 -- 172.21.15.84:0/3862013000 <== mon.1 v1:172.21.15.179:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 244792291 0 0) 0x7fb7c806fea0 con 0x7fb7c806dd90 2024-03-29T11:25:57.323 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:57.327+0000 7fb7c5ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:25:57.323 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:57.327+0000 7fb7c5ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:25:57.323 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:57.327+0000 7fb7c5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:25:57.323 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:57.327+0000 7fb7c5ffb640 10 monclient(hunting): my global_id is 4426 2024-03-29T11:25:57.323 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:57.327+0000 7fb7c5ffb640 1 -- 172.21.15.84:0/3862013000 --> v1:172.21.15.179:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb7b4004eb0 con 0x7fb7c806dd90 2024-03-29T11:25:57.323 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:57.327+0000 7fb7c5ffb640 1 -- 172.21.15.84:0/3862013000 <== mon.0 v1:172.21.15.84:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1946838497 0 0) 0x7fb7c806fc30 con 0x7fb7c80aeca0 2024-03-29T11:25:57.323 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:57.327+0000 7fb7c5ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:25:57.323 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:57.327+0000 7fb7c5ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:25:57.323 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:57.327+0000 7fb7c5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:25:57.323 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:57.327+0000 7fb7c5ffb640 10 monclient(hunting): my global_id is 4464 2024-03-29T11:25:57.323 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:57.327+0000 7fb7c5ffb640 1 -- 172.21.15.84:0/3862013000 --> v1:172.21.15.84:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb7c806fea0 con 0x7fb7c80aeca0 2024-03-29T11:25:57.323 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:57.327+0000 7fb7c5ffb640 1 -- 172.21.15.84:0/3862013000 <== mon.1 v1:172.21.15.179:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3048255401 0 0) 0x7fb7b4004eb0 con 0x7fb7c806dd90 2024-03-29T11:25:57.323 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:57.328+0000 7fb7c5ffb640 1 -- 172.21.15.84:0/3862013000 --> v1:172.21.15.179:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb7c806fc30 con 0x7fb7c806dd90 2024-03-29T11:25:57.323 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:57.328+0000 7fb7c5ffb640 1 -- 172.21.15.84:0/3862013000 <== mon.2 v1:172.21.15.84:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3102881487 0 0) 0x7fb7c8070180 con 0x7fb7c80b0500 2024-03-29T11:25:57.324 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:57.328+0000 7fb7c5ffb640 1 -- 172.21.15.84:0/3862013000 >> v1:172.21.15.179:6789/0 conn(0x7fb7c806dd90 legacy=0x7fb7c81e61a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:57.324 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:57.328+0000 7fb7c5ffb640 1 -- 172.21.15.84:0/3862013000 >> v1:172.21.15.84:6789/0 conn(0x7fb7c80aeca0 legacy=0x7fb7c806e6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:57.324 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:57.328+0000 7fb7c5ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:25:57.324 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:57.328+0000 7fb7c5ffb640 1 monclient: found mon.c 2024-03-29T11:25:57.324 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:57.328+0000 7fb7c5ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:25:57.324 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:57.328+0000 7fb7c5ffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-29T11:25:57.324 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:57.328+0000 7fb7c5ffb640 1 -- 172.21.15.84:0/3862013000 --> v1:172.21.15.84:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7c8070e10 con 0x7fb7c80b0500 2024-03-29T11:25:57.324 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:57.328+0000 7fb7c5ffb640 10 monclient: _finish_auth 0 2024-03-29T11:25:57.324 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:57.328+0000 7fb7c5ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:25:57.324 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:57.328+0000 7fb7c5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:25:57.324 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:57.328+0000 7fb7cf688640 5 monclient: authenticate success, global_id 4478 2024-03-29T11:25:57.324 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:57.328+0000 7fb7cf688640 10 monclient: _renew_subs 2024-03-29T11:25:57.324 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:57.328+0000 7fb7cf688640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-29T11:25:57.324 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:57.328+0000 7fb7cf688640 1 -- 172.21.15.84:0/3862013000 --> v1:172.21.15.84:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb7c8070880 con 0x7fb7c80b0500 2024-03-29T11:25:57.324 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:57.328+0000 7fb7cf688640 10 client.4478.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:25:57.325 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:57.328+0000 7fb7cf688640 10 monclient: _renew_subs 2024-03-29T11:25:57.325 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:57.328+0000 7fb7cf688640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-29T11:25:57.325 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:57.328+0000 7fb7cf688640 1 -- 172.21.15.84:0/3862013000 --> v1:172.21.15.84:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb7c81e78c0 con 0x7fb7c80b0500 2024-03-29T11:25:57.325 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:57.328+0000 7fb7cf688640 1 librados: init done 2024-03-29T11:25:57.325 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:57.328+0000 7fb7c5ffb640 1 -- 172.21.15.84:0/3862013000 <== mon.2 v1:172.21.15.84:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb7b8001c50 con 0x7fb7c80b0500 2024-03-29T11:25:57.325 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:57.328+0000 7fb7c5ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:25:57.325 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:57.328+0000 7fb7c77fe640 4 set_mon_vals no callback set 2024-03-29T11:25:57.325 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:57.329+0000 7fb7af7fe640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:25:57.325 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:57.330+0000 7fb7c5ffb640 1 -- 172.21.15.84:0/3862013000 <== mon.2 v1:172.21.15.84:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4236868694 0 0) 0x7fb7b8003790 con 0x7fb7c80b0500 2024-03-29T11:25:57.325 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:57.330+0000 7fb7af7fe640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:25:57.325 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:57.330+0000 7fb7af7fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-29T11:25:57.325 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:57.330+0000 7fb7af7fe640 1 -- 172.21.15.84:0/3862013000 --> v1:172.21.15.84:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb7c80ad0c0 con 0x7fb7c80b0500 2024-03-29T11:25:57.325 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:57.330+0000 7fb7c5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:25:57.325 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:57.330+0000 7fb7c5ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:25:57.325 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:57.330+0000 7fb7c5ffb640 10 monclient: dump: 2024-03-29T11:25:57.325 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T11:25:57.326 INFO:teuthology.misc.health.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:25:57.326 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T11:18:53.864025+0000 2024-03-29T11:25:57.326 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T11:18:53.864025+0000 2024-03-29T11:25:57.326 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T11:25:57.326 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:25:57.326 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T11:25:57.326 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T11:25:57.326 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T11:25:57.326 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:25:57.326 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:57.331+0000 7fb7c5ffb640 1 -- 172.21.15.84:0/3862013000 <== mon.2 v1:172.21.15.84:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4033942909 0 0) 0x7fb7b8019760 con 0x7fb7c80b0500 2024-03-29T11:25:57.326 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:57.331+0000 7fb7c5ffb640 1 -- 172.21.15.84:0/3862013000 <== mon.2 v1:172.21.15.84:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2808139587 0 0) 0x7fb7b8083620 con 0x7fb7c80b0500 2024-03-29T11:25:57.326 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:57.331+0000 7fb7cd424640 10 monclient: get_auth_request con 0x7fb7b406ded0 auth_method 0 2024-03-29T11:25:57.326 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:57.331+0000 7fb7c5ffb640 10 client.4478.objecter ms_dispatch 0x7fb7c8000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:25:57.326 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:57.331+0000 7fb7c5ffb640 3 client.4478.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:25:57.326 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:57.331+0000 7fb7c5ffb640 3 client.4478.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:25:57.327 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:57.331+0000 7fb7c5ffb640 20 client.4478.objecter dump_active .. 0 homeless 2024-03-29T11:25:57.327 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:57.332+0000 7fb7c5ffb640 10 client.4478.objecter ms_handle_connect 0x7fb7b406ded0 2024-03-29T11:25:57.327 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:57.341+0000 7fb7c5ffb640 1 -- 172.21.15.84:0/3862013000 <== mon.2 v1:172.21.15.84:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fb7b804ef00 con 0x7fb7c80b0500 2024-03-29T11:25:57.327 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:57.341+0000 7fb7c5ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:25:57.327 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:57.341+0000 7fb7c5ffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:25:57.675 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:57.690+0000 7fb7af7fe640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:25:57.675 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:57.690+0000 7fb7af7fe640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:25:57.675 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:57.690+0000 7fb7af7fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-29T11:25:57.675 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:57.690+0000 7fb7af7fe640 1 -- 172.21.15.84:0/3862013000 --> v1:172.21.15.84:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb7c8075f10 con 0x7fb7c80b0500 2024-03-29T11:25:57.676 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:57.691+0000 7fb7c5ffb640 1 -- 172.21.15.84:0/3862013000 <== mon.2 v1:172.21.15.84:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4209477549) 0x7fb7b8050e30 con 0x7fb7c80b0500 2024-03-29T11:25:57.676 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:57.691+0000 7fb7c5ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:25:57.677 INFO:teuthology.misc.health.smithi084.stdout: 2024-03-29T11:25:57.677 INFO:teuthology.misc.health.smithi084.stdout:{"epoch":10,"fsid":"2c3e002f-9eb0-4860-853b-4ef1ef562213","created":"2024-03-29T11:19:32.029169+0000","modified":"2024-03-29T11:19:34.200299+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:34.200299+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"5c39e276-2034-4716-bbc0-57a634eefa92","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"cd580e99-109f-4bf1-88c0-a5987bc8b9ed","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"11e60a5a-051d-453f-a190-66387cb35592","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"2c329fbe-bbc4-4030-bf87-d711cad04679","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f4213d4a-bd36-4433-91ae-d7e2b02481d3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"f0cf44ea-6055-48af-805b-aed858102ff9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"d895eaa0-cc0b-4ee4-9b5a-2dfcd89a60b7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"9bc5a7bc-8f52-4db9-96d6-de5e68d52eaa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:25:57.677 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:57.691+0000 7fb7c5ffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:25:57.680 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:57.695+0000 7fb7af7fe640 10 librados: watch_flush enter 2024-03-29T11:25:57.681 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:57.695+0000 7fb7af7fe640 10 librados: watch_flush exit 2024-03-29T11:25:57.681 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:57.695+0000 7fb7af7fe640 20 client.4478.objecter shutdown clearing up homeless session... 2024-03-29T11:25:57.681 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:57.695+0000 7fb7af7fe640 10 client.4478.objecter successfully canceled tick 2024-03-29T11:25:57.681 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:57.695+0000 7fb7af7fe640 1 -- 172.21.15.84:0/3862013000 >> v1:172.21.15.179:6816/2981134980 conn(0x7fb7b406ded0 legacy=0x7fb7b4070390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:57.681 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:57.695+0000 7fb7af7fe640 10 monclient: shutdown 2024-03-29T11:25:57.681 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:57.695+0000 7fb7af7fe640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:25:57.681 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:57.695+0000 7fb7af7fe640 1 -- 172.21.15.84:0/3862013000 >> v1:172.21.15.84:6790/0 conn(0x7fb7c80b0500 legacy=0x7fb7c806edf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:57.681 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:57.695+0000 7fb7af7fe640 1 -- 172.21.15.84:0/3862013000 shutdown_connections 2024-03-29T11:25:57.681 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:57.695+0000 7fb7af7fe640 1 -- 172.21.15.84:0/3862013000 >> 172.21.15.84:0/3862013000 conn(0x7fb7c806b660 msgr2=0x7fb7c806ba40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:25:57.681 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:57.696+0000 7fb7af7fe640 1 -- 172.21.15.84:0/3862013000 shutdown_connections 2024-03-29T11:25:57.681 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:57.696+0000 7fb7af7fe640 1 -- 172.21.15.84:0/3862013000 wait complete. 2024-03-29T11:25:57.681 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:25:57.696+0000 7fb7af7fe640 1 librados: shutdown 2024-03-29T11:25:57.696 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:26:04.503 DEBUG:teuthology.orchestra.run.smithi084:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:26:04.657 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:04.670+0000 7f7560c4e640 10 monclient: get_monmap_and_config 2024-03-29T11:26:04.657 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:04.670+0000 7f7560c4e640 10 monclient: build_initial_monmap 2024-03-29T11:26:04.657 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:04.670+0000 7f7560c4e640 10 monclient: monmap: 2024-03-29T11:26:04.657 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-29T11:26:04.657 INFO:teuthology.misc.health.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:26:04.657 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T11:26:04.671817+0000 2024-03-29T11:26:04.657 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T11:26:04.671817+0000 2024-03-29T11:26:04.657 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-29T11:26:04.657 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:26:04.657 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-29T11:26:04.657 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.noname-c 2024-03-29T11:26:04.658 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-b 2024-03-29T11:26:04.658 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:26:04.658 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:04.672+0000 7f7560c4e640 1 Processor -- start 2024-03-29T11:26:04.658 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:04.672+0000 7f7560c4e640 1 -- start start 2024-03-29T11:26:04.658 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:04.672+0000 7f7560c4e640 10 monclient: init 2024-03-29T11:26:04.661 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:04.673+0000 7f7560c4e640 10 monclient: _reopen_session rank -1 2024-03-29T11:26:04.662 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:04.673+0000 7f7560c4e640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:26:04.662 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:04.673+0000 7f7560c4e640 10 monclient(hunting): picked mon.noname-b con 0x7f755c0a9930 addr v1:172.21.15.84:6790/0 2024-03-29T11:26:04.662 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:04.673+0000 7f7560c4e640 10 monclient(hunting): picked mon.noname-c con 0x7f755c149dc0 addr v1:172.21.15.179:6789/0 2024-03-29T11:26:04.663 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:04.673+0000 7f7560c4e640 10 monclient(hunting): picked mon.noname-a con 0x7f755c14ff40 addr v1:172.21.15.84:6789/0 2024-03-29T11:26:04.663 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:04.673+0000 7f7560c4e640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f755c0aae60 con 0x7f755c14ff40 2024-03-29T11:26:04.663 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:04.673+0000 7f7560c4e640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f755c0b4310 con 0x7f755c149dc0 2024-03-29T11:26:04.664 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:04.673+0000 7f7560c4e640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f755c0afdd0 con 0x7f755c0a9930 2024-03-29T11:26:04.664 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:04.673+0000 7f7560c4e640 10 monclient(hunting): _renew_subs 2024-03-29T11:26:04.664 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:04.673+0000 7f7560c4e640 10 monclient(hunting): authenticate will time out at 1225.967666s 2024-03-29T11:26:04.664 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:04.678+0000 7f755a59c640 1 --1- >> v1:172.21.15.84:6790/0 conn(0x7f755c0a9930 0x7f755c143a30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.84:6790/0 says I am v1:172.21.15.84:47700/0 (socket says 172.21.15.84:47700) 2024-03-29T11:26:04.664 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:04.678+0000 7f755a59c640 1 -- 172.21.15.84:0/3582579722 learned_addr learned my addr 172.21.15.84:0/3582579722 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-29T11:26:04.666 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:04.681+0000 7f755959a640 1 -- 172.21.15.84:0/3582579722 <== mon.0 v1:172.21.15.84:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 895362042 0 0) 0x7f755c0aae60 con 0x7f755c14ff40 2024-03-29T11:26:04.666 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:04.681+0000 7f755959a640 10 monclient(hunting): _negotiate 2024-03-29T11:26:04.666 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:04.681+0000 7f755959a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:26:04.666 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:04.681+0000 7f755959a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:26:04.666 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:04.681+0000 7f755959a640 10 monclient(hunting): my global_id is 4467 2024-03-29T11:26:04.666 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:04.681+0000 7f755959a640 1 -- 172.21.15.84:0/3582579722 --> v1:172.21.15.84:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7544003520 con 0x7f755c14ff40 2024-03-29T11:26:04.666 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:04.681+0000 7f755959a640 1 -- 172.21.15.84:0/3582579722 <== mon.1 v1:172.21.15.179:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1877987420 0 0) 0x7f755c0b4310 con 0x7f755c149dc0 2024-03-29T11:26:04.667 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:04.681+0000 7f755959a640 10 monclient(hunting): _negotiate 2024-03-29T11:26:04.667 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:04.681+0000 7f755959a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:26:04.667 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:04.681+0000 7f755959a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:26:04.667 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:04.681+0000 7f755959a640 10 monclient(hunting): my global_id is 4429 2024-03-29T11:26:04.667 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:04.681+0000 7f755959a640 1 -- 172.21.15.84:0/3582579722 --> v1:172.21.15.179:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f755c0aae60 con 0x7f755c149dc0 2024-03-29T11:26:04.667 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:04.681+0000 7f755959a640 1 -- 172.21.15.84:0/3582579722 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3211376947 0 0) 0x7f755c0afdd0 con 0x7f755c0a9930 2024-03-29T11:26:04.667 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:04.681+0000 7f755959a640 10 monclient(hunting): _negotiate 2024-03-29T11:26:04.667 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:04.681+0000 7f755959a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:26:04.667 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:04.681+0000 7f755959a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:26:04.667 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:04.681+0000 7f755959a640 10 monclient(hunting): my global_id is 4481 2024-03-29T11:26:04.667 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:04.681+0000 7f755959a640 1 -- 172.21.15.84:0/3582579722 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f755c0b4310 con 0x7f755c0a9930 2024-03-29T11:26:04.668 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:04.682+0000 7f755959a640 1 -- 172.21.15.84:0/3582579722 <== mon.1 v1:172.21.15.179:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3723968303 0 0) 0x7f755c0aae60 con 0x7f755c149dc0 2024-03-29T11:26:04.668 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:04.682+0000 7f755959a640 1 -- 172.21.15.84:0/3582579722 >> v1:172.21.15.84:6790/0 conn(0x7f755c0a9930 legacy=0x7f755c143a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:04.668 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:04.682+0000 7f755959a640 1 -- 172.21.15.84:0/3582579722 >> v1:172.21.15.84:6789/0 conn(0x7f755c14ff40 legacy=0x7f755c152330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:04.668 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:04.682+0000 7f755959a640 10 monclient: _finish_hunting 0 2024-03-29T11:26:04.668 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:04.682+0000 7f755959a640 1 monclient: found mon.noname-c 2024-03-29T11:26:04.668 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:04.682+0000 7f755959a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:26:04.669 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:04.682+0000 7f755959a640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.179:6789/0 2024-03-29T11:26:04.669 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:04.682+0000 7f755959a640 1 -- 172.21.15.84:0/3582579722 --> v1:172.21.15.179:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f755c14a950 con 0x7f755c149dc0 2024-03-29T11:26:04.669 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:04.682+0000 7f755959a640 10 monclient: _finish_auth 0 2024-03-29T11:26:04.669 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:04.682+0000 7f755959a640 10 monclient: _check_auth_tickets 2024-03-29T11:26:04.669 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:04.682+0000 7f755959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:26:04.669 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:04.682+0000 7f755959a640 1 -- 172.21.15.84:0/3582579722 <== mon.1 v1:172.21.15.179:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4236868694 0 0) 0x7f7550001520 con 0x7f755c149dc0 2024-03-29T11:26:04.669 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:04.682+0000 7f755959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:04.669 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:04.682+0000 7f755959a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:26:04.669 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:04.682+0000 7f755959a640 10 monclient: dump: 2024-03-29T11:26:04.669 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T11:26:04.669 INFO:teuthology.misc.health.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:26:04.669 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T11:18:53.864025+0000 2024-03-29T11:26:04.669 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T11:18:53.864025+0000 2024-03-29T11:26:04.669 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:04.670 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:26:04.670 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T11:26:04.670 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T11:26:04.670 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T11:26:04.670 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:26:04.670 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:04.682+0000 7f7560c4e640 5 monclient: authenticate success, global_id 4429 2024-03-29T11:26:04.670 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:04.682+0000 7f7560c4e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:26:04.670 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:04.682+0000 7f755959a640 1 -- 172.21.15.84:0/3582579722 <== mon.1 v1:172.21.15.179:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7550001680 con 0x7f755c149dc0 2024-03-29T11:26:04.670 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:04.682+0000 7f755959a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:26:04.670 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:04.682+0000 7f755959a640 1 -- 172.21.15.84:0/3582579722 <== mon.1 v1:172.21.15.179:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4236868694 0 0) 0x7f7550001ad0 con 0x7f755c149dc0 2024-03-29T11:26:04.670 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:04.682+0000 7f755959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:04.670 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:04.682+0000 7f755959a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:26:04.670 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:04.682+0000 7f755959a640 10 monclient: dump: 2024-03-29T11:26:04.670 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T11:26:04.670 INFO:teuthology.misc.health.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:26:04.671 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T11:18:53.864025+0000 2024-03-29T11:26:04.671 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T11:18:53.864025+0000 2024-03-29T11:26:04.671 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:04.671 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:26:04.671 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T11:26:04.671 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T11:26:04.671 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T11:26:04.671 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:26:04.672 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:04.683+0000 7f7560c4e640 10 monclient: get_monmap_and_config success 2024-03-29T11:26:04.672 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:04.683+0000 7f7560c4e640 4 set_mon_vals no callback set 2024-03-29T11:26:04.672 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:04.684+0000 7f7560c4e640 10 monclient: shutdown 2024-03-29T11:26:04.672 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:04.684+0000 7f7560c4e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:26:04.672 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:04.684+0000 7f7560c4e640 1 -- 172.21.15.84:0/3582579722 >> v1:172.21.15.179:6789/0 conn(0x7f755c149dc0 legacy=0x7f755c14a240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:04.672 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:04.684+0000 7f7560c4e640 1 -- 172.21.15.84:0/3582579722 shutdown_connections 2024-03-29T11:26:04.672 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:04.684+0000 7f7560c4e640 1 -- 172.21.15.84:0/3582579722 >> 172.21.15.84:0/3582579722 conn(0x7f755c0b35e0 msgr2=0x7f755c0b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:26:04.673 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:04.686+0000 7f7560c4e640 1 -- 172.21.15.84:0/3582579722 shutdown_connections 2024-03-29T11:26:04.673 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:04.686+0000 7f7560c4e640 1 -- 172.21.15.84:0/3582579722 wait complete. 2024-03-29T11:26:04.673 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:04.687+0000 7f7560c4e640 10 monclient: build_initial_monmap 2024-03-29T11:26:04.673 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:04.687+0000 7f7560c4e640 10 monclient: monmap: 2024-03-29T11:26:04.673 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-29T11:26:04.673 INFO:teuthology.misc.health.smithi084.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:26:04.674 INFO:teuthology.misc.health.smithi084.stderr:last_changed 0.000000 2024-03-29T11:26:04.674 INFO:teuthology.misc.health.smithi084.stderr:created 0.000000 2024-03-29T11:26:04.674 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-29T11:26:04.674 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:26:04.674 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-29T11:26:04.674 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.noname-b 2024-03-29T11:26:04.674 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-c 2024-03-29T11:26:04.674 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:26:04.674 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:04.687+0000 7f7560c4e640 1 librados: starting msgr at 2024-03-29T11:26:04.674 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:04.687+0000 7f7560c4e640 1 librados: starting objecter 2024-03-29T11:26:04.674 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:04.687+0000 7f7560c4e640 1 Processor -- start 2024-03-29T11:26:04.674 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:04.687+0000 7f7560c4e640 1 -- start start 2024-03-29T11:26:04.674 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:04.687+0000 7f7560c4e640 1 librados: setting wanted keys 2024-03-29T11:26:04.674 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:04.687+0000 7f7560c4e640 1 librados: calling monclient init 2024-03-29T11:26:04.675 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:04.687+0000 7f7560c4e640 10 monclient: init 2024-03-29T11:26:04.676 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:04.688+0000 7f7560c4e640 10 monclient: _reopen_session rank -1 2024-03-29T11:26:04.677 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:04.688+0000 7f7560c4e640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:26:04.677 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:04.688+0000 7f7560c4e640 10 monclient(hunting): picked mon.noname-a con 0x7f755c0a9930 addr v1:172.21.15.84:6789/0 2024-03-29T11:26:04.677 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:04.688+0000 7f7560c4e640 10 monclient(hunting): picked mon.noname-b con 0x7f755c149dc0 addr v1:172.21.15.179:6789/0 2024-03-29T11:26:04.677 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:04.688+0000 7f7560c4e640 10 monclient(hunting): picked mon.noname-c con 0x7f755c14ff40 addr v1:172.21.15.84:6790/0 2024-03-29T11:26:04.677 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:04.688+0000 7f7560c4e640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f755c13dd40 con 0x7f755c0a9930 2024-03-29T11:26:04.677 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:04.688+0000 7f7560c4e640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f755c147530 con 0x7f755c149dc0 2024-03-29T11:26:04.677 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:04.688+0000 7f7560c4e640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f755c144510 con 0x7f755c14ff40 2024-03-29T11:26:04.677 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:04.688+0000 7f7560c4e640 10 monclient(hunting): _renew_subs 2024-03-29T11:26:04.677 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:04.688+0000 7f7560c4e640 10 monclient(hunting): authenticate will time out at 1225.983029s 2024-03-29T11:26:04.678 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:04.692+0000 7f755a59c640 1 --1- >> v1:172.21.15.84:6789/0 conn(0x7f755c0a9930 0x7f755c13bf00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.84:6789/0 says I am v1:172.21.15.84:46091/0 (socket says 172.21.15.84:46091) 2024-03-29T11:26:04.678 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:04.692+0000 7f755a59c640 1 -- 172.21.15.84:0/3651786271 learned_addr learned my addr 172.21.15.84:0/3651786271 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-29T11:26:04.679 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:04.694+0000 7f754affd640 10 client.?.objecter ms_handle_connect 0x7f755c14ff40 2024-03-29T11:26:04.679 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:04.694+0000 7f754affd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:26:04.679 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:04.694+0000 7f754affd640 10 client.?.objecter ms_handle_connect 0x7f755c0a9930 2024-03-29T11:26:04.679 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:04.694+0000 7f754affd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:26:04.679 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:04.694+0000 7f754affd640 1 -- 172.21.15.84:0/3651786271 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1147050504 0 0) 0x7f755c144510 con 0x7f755c14ff40 2024-03-29T11:26:04.679 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:04.694+0000 7f754affd640 10 monclient(hunting): _negotiate 2024-03-29T11:26:04.679 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:04.694+0000 7f754affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:26:04.679 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:04.694+0000 7f754affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:26:04.679 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:04.694+0000 7f754affd640 10 monclient(hunting): my global_id is 4484 2024-03-29T11:26:04.679 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:04.694+0000 7f754affd640 1 -- 172.21.15.84:0/3651786271 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f753c004eb0 con 0x7f755c14ff40 2024-03-29T11:26:04.680 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:04.694+0000 7f754affd640 1 -- 172.21.15.84:0/3651786271 <== mon.0 v1:172.21.15.84:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3488116187 0 0) 0x7f755c13dd40 con 0x7f755c0a9930 2024-03-29T11:26:04.680 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:04.694+0000 7f754affd640 10 monclient(hunting): _negotiate 2024-03-29T11:26:04.680 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:04.694+0000 7f754affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:26:04.680 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:04.694+0000 7f754affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:26:04.680 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:04.694+0000 7f754affd640 10 monclient(hunting): my global_id is 4470 2024-03-29T11:26:04.680 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:04.694+0000 7f754affd640 1 -- 172.21.15.84:0/3651786271 --> v1:172.21.15.84:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f755c144510 con 0x7f755c0a9930 2024-03-29T11:26:04.681 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:04.694+0000 7f754affd640 1 -- 172.21.15.84:0/3651786271 <== mon.0 v1:172.21.15.84:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 646967702 0 0) 0x7f755c144510 con 0x7f755c0a9930 2024-03-29T11:26:04.681 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:04.694+0000 7f754affd640 1 -- 172.21.15.84:0/3651786271 --> v1:172.21.15.84:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f755c13dd40 con 0x7f755c0a9930 2024-03-29T11:26:04.681 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:04.694+0000 7f754affd640 1 -- 172.21.15.84:0/3651786271 <== mon.0 v1:172.21.15.84:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4236868694 0 0) 0x7f754c001870 con 0x7f755c0a9930 2024-03-29T11:26:04.681 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:04.694+0000 7f754affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:04.681 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:04.694+0000 7f754affd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:26:04.681 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:04.694+0000 7f754affd640 10 monclient(hunting): dump: 2024-03-29T11:26:04.682 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T11:26:04.682 INFO:teuthology.misc.health.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:26:04.682 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T11:18:53.864025+0000 2024-03-29T11:26:04.682 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T11:18:53.864025+0000 2024-03-29T11:26:04.682 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:04.682 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:26:04.682 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T11:26:04.682 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T11:26:04.682 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T11:26:04.682 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:26:04.682 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:04.694+0000 7f754affd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:26:04.682 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:04.694+0000 7f754affd640 1 -- 172.21.15.84:0/3651786271 <== mon.2 v1:172.21.15.84:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 858526647 0 0) 0x7f753c004eb0 con 0x7f755c14ff40 2024-03-29T11:26:04.682 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:04.694+0000 7f754affd640 1 -- 172.21.15.84:0/3651786271 --> v1:172.21.15.84:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f755c144510 con 0x7f755c14ff40 2024-03-29T11:26:04.682 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:04.695+0000 7f754affd640 1 -- 172.21.15.84:0/3651786271 <== mon.2 v1:172.21.15.84:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4236868694 0 0) 0x7f7554001c10 con 0x7f755c14ff40 2024-03-29T11:26:04.683 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:04.695+0000 7f754affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:04.683 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:04.695+0000 7f754affd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:26:04.683 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:04.695+0000 7f754affd640 10 monclient(hunting): dump: 2024-03-29T11:26:04.683 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T11:26:04.683 INFO:teuthology.misc.health.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:26:04.683 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T11:18:53.864025+0000 2024-03-29T11:26:04.683 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T11:18:53.864025+0000 2024-03-29T11:26:04.683 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:04.683 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:26:04.683 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T11:26:04.683 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T11:26:04.683 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T11:26:04.683 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:26:04.683 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:04.695+0000 7f754affd640 10 client.?.objecter ms_handle_connect 0x7f755c149dc0 2024-03-29T11:26:04.683 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:04.695+0000 7f754affd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:26:04.684 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:04.695+0000 7f754affd640 1 -- 172.21.15.84:0/3651786271 <== mon.2 v1:172.21.15.84:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3841285105 0 0) 0x7f755c144510 con 0x7f755c14ff40 2024-03-29T11:26:04.684 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:04.695+0000 7f754affd640 1 -- 172.21.15.84:0/3651786271 >> v1:172.21.15.179:6789/0 conn(0x7f755c149dc0 legacy=0x7f755c147c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:04.684 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:04.695+0000 7f754affd640 1 -- 172.21.15.84:0/3651786271 >> v1:172.21.15.84:6789/0 conn(0x7f755c0a9930 legacy=0x7f755c13bf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:04.684 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:04.695+0000 7f754affd640 10 monclient: _finish_hunting 0 2024-03-29T11:26:04.684 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:04.695+0000 7f754affd640 1 monclient: found mon.c 2024-03-29T11:26:04.684 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:04.695+0000 7f754affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:26:04.684 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:04.695+0000 7f754affd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-29T11:26:04.684 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:04.695+0000 7f754affd640 1 -- 172.21.15.84:0/3651786271 --> v1:172.21.15.84:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f755c1463a0 con 0x7f755c14ff40 2024-03-29T11:26:04.684 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:04.695+0000 7f754affd640 10 monclient: _finish_auth 0 2024-03-29T11:26:04.684 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:04.695+0000 7f754affd640 10 monclient: _check_auth_tickets 2024-03-29T11:26:04.684 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:04.695+0000 7f754affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:26:04.684 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:04.695+0000 7f7560c4e640 5 monclient: authenticate success, global_id 4484 2024-03-29T11:26:04.684 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:04.695+0000 7f7560c4e640 10 monclient: _renew_subs 2024-03-29T11:26:04.684 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:04.695+0000 7f7560c4e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-29T11:26:04.684 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:04.695+0000 7f7560c4e640 1 -- 172.21.15.84:0/3651786271 --> v1:172.21.15.84:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f755c13d790 con 0x7f755c14ff40 2024-03-29T11:26:04.685 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:04.695+0000 7f7560c4e640 10 client.4484.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:26:04.685 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:04.695+0000 7f7560c4e640 10 monclient: _renew_subs 2024-03-29T11:26:04.685 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:04.695+0000 7f7560c4e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-29T11:26:04.685 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:04.695+0000 7f7560c4e640 1 -- 172.21.15.84:0/3651786271 --> v1:172.21.15.84:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f755c13da80 con 0x7f755c14ff40 2024-03-29T11:26:04.685 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:04.695+0000 7f7560c4e640 1 librados: init done 2024-03-29T11:26:04.685 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:04.697+0000 7f7560c4e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:26:04.685 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:04.697+0000 7f7560c4e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:26:04.685 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:04.697+0000 7f7560c4e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-29T11:26:04.685 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:04.697+0000 7f7560c4e640 1 -- 172.21.15.84:0/3651786271 --> v1:172.21.15.84:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f755c075f10 con 0x7f755c14ff40 2024-03-29T11:26:04.686 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:04.699+0000 7f754affd640 1 -- 172.21.15.84:0/3651786271 <== mon.2 v1:172.21.15.84:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7554003990 con 0x7f755c14ff40 2024-03-29T11:26:04.686 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:04.699+0000 7f754affd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:26:04.686 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:04.699+0000 7f754affd640 1 -- 172.21.15.84:0/3651786271 <== mon.2 v1:172.21.15.84:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4236868694 0 0) 0x7f7554001d70 con 0x7f755c14ff40 2024-03-29T11:26:04.686 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:04.699+0000 7f754affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:04.686 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:04.699+0000 7f754affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:26:04.686 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:04.699+0000 7f754affd640 10 monclient: dump: 2024-03-29T11:26:04.686 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T11:26:04.686 INFO:teuthology.misc.health.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:26:04.686 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T11:18:53.864025+0000 2024-03-29T11:26:04.686 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T11:18:53.864025+0000 2024-03-29T11:26:04.686 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:04.686 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:26:04.686 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T11:26:04.686 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T11:26:04.686 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T11:26:04.687 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:26:04.687 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:04.699+0000 7f754bfff640 4 set_mon_vals no callback set 2024-03-29T11:26:04.688 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:04.700+0000 7f754affd640 1 -- 172.21.15.84:0/3651786271 <== mon.2 v1:172.21.15.84:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4033942909 0 0) 0x7f7554019b10 con 0x7f755c14ff40 2024-03-29T11:26:04.688 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:04.701+0000 7f754affd640 1 -- 172.21.15.84:0/3651786271 <== mon.2 v1:172.21.15.84:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2808139587 0 0) 0x7f7554083a80 con 0x7f755c14ff40 2024-03-29T11:26:04.688 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:04.701+0000 7f754affd640 10 client.4484.objecter ms_dispatch 0x7f755c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:26:04.688 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:04.701+0000 7f754affd640 3 client.4484.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:26:04.688 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:04.701+0000 7f754affd640 3 client.4484.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:26:04.688 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:04.701+0000 7f754affd640 20 client.4484.objecter dump_active .. 0 homeless 2024-03-29T11:26:04.688 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:04.701+0000 7f755a59c640 10 monclient: get_auth_request con 0x7f753c06df60 auth_method 0 2024-03-29T11:26:04.688 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:04.702+0000 7f754affd640 10 client.4484.objecter ms_handle_connect 0x7f753c06df60 2024-03-29T11:26:04.696 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:04.711+0000 7f754affd640 1 -- 172.21.15.84:0/3651786271 <== mon.2 v1:172.21.15.84:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f755404f190 con 0x7f755c14ff40 2024-03-29T11:26:04.696 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:04.711+0000 7f754affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:26:04.696 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:04.711+0000 7f754affd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:26:05.031 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:05.045+0000 7f7560c4e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:26:05.031 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:05.045+0000 7f7560c4e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:26:05.031 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:05.045+0000 7f7560c4e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-29T11:26:05.031 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:05.045+0000 7f7560c4e640 1 -- 172.21.15.84:0/3651786271 --> v1:172.21.15.84:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f755c0ad2f0 con 0x7f755c14ff40 2024-03-29T11:26:05.034 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:05.049+0000 7f754affd640 1 -- 172.21.15.84:0/3651786271 <== mon.2 v1:172.21.15.84:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4209477549) 0x7f7554051290 con 0x7f755c14ff40 2024-03-29T11:26:05.034 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:05.049+0000 7f754affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:26:05.034 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:05.049+0000 7f754affd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:26:05.035 INFO:teuthology.misc.health.smithi084.stdout: 2024-03-29T11:26:05.035 INFO:teuthology.misc.health.smithi084.stdout:{"epoch":10,"fsid":"2c3e002f-9eb0-4860-853b-4ef1ef562213","created":"2024-03-29T11:19:32.029169+0000","modified":"2024-03-29T11:19:34.200299+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:34.200299+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"5c39e276-2034-4716-bbc0-57a634eefa92","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"cd580e99-109f-4bf1-88c0-a5987bc8b9ed","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"11e60a5a-051d-453f-a190-66387cb35592","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"2c329fbe-bbc4-4030-bf87-d711cad04679","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f4213d4a-bd36-4433-91ae-d7e2b02481d3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"f0cf44ea-6055-48af-805b-aed858102ff9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"d895eaa0-cc0b-4ee4-9b5a-2dfcd89a60b7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"9bc5a7bc-8f52-4db9-96d6-de5e68d52eaa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:26:05.039 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:05.053+0000 7f7560c4e640 10 librados: watch_flush enter 2024-03-29T11:26:05.039 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:05.053+0000 7f7560c4e640 10 librados: watch_flush exit 2024-03-29T11:26:05.039 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:05.053+0000 7f7560c4e640 20 client.4484.objecter shutdown clearing up homeless session... 2024-03-29T11:26:05.039 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:05.053+0000 7f7560c4e640 10 client.4484.objecter successfully canceled tick 2024-03-29T11:26:05.039 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:05.053+0000 7f7560c4e640 1 -- 172.21.15.84:0/3651786271 >> v1:172.21.15.179:6816/2981134980 conn(0x7f753c06df60 legacy=0x7f753c070420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:05.039 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:05.053+0000 7f7560c4e640 10 monclient: shutdown 2024-03-29T11:26:05.040 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:05.053+0000 7f7560c4e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:26:05.040 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:05.053+0000 7f7560c4e640 1 -- 172.21.15.84:0/3651786271 >> v1:172.21.15.84:6790/0 conn(0x7f755c14ff40 legacy=0x7f755c2000a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:05.040 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:05.053+0000 7f7560c4e640 1 -- 172.21.15.84:0/3651786271 shutdown_connections 2024-03-29T11:26:05.040 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:05.053+0000 7f7560c4e640 1 -- 172.21.15.84:0/3651786271 >> 172.21.15.84:0/3651786271 conn(0x7f755c0b35e0 msgr2=0x7f755c0b39c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:26:05.040 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:05.054+0000 7f7560c4e640 1 -- 172.21.15.84:0/3651786271 shutdown_connections 2024-03-29T11:26:05.040 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:05.054+0000 7f7560c4e640 1 -- 172.21.15.84:0/3651786271 wait complete. 2024-03-29T11:26:05.040 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:05.054+0000 7f7560c4e640 1 librados: shutdown 2024-03-29T11:26:05.056 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:26:11.863 DEBUG:teuthology.orchestra.run.smithi084:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:26:12.005 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:12.019+0000 7f98b727b640 10 monclient: get_monmap_and_config 2024-03-29T11:26:12.005 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:12.019+0000 7f98b727b640 10 monclient: build_initial_monmap 2024-03-29T11:26:12.005 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:12.019+0000 7f98b727b640 10 monclient: monmap: 2024-03-29T11:26:12.005 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-29T11:26:12.005 INFO:teuthology.misc.health.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:26:12.006 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T11:26:12.020706+0000 2024-03-29T11:26:12.006 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T11:26:12.020706+0000 2024-03-29T11:26:12.006 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-29T11:26:12.006 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:26:12.006 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-29T11:26:12.006 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.noname-c 2024-03-29T11:26:12.006 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-b 2024-03-29T11:26:12.006 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:26:12.006 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:12.020+0000 7f98b727b640 1 Processor -- start 2024-03-29T11:26:12.007 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:12.021+0000 7f98b727b640 1 -- start start 2024-03-29T11:26:12.007 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:12.021+0000 7f98b727b640 10 monclient: init 2024-03-29T11:26:12.009 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:12.022+0000 7f98b727b640 10 monclient: _reopen_session rank -1 2024-03-29T11:26:12.009 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:12.022+0000 7f98b727b640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:26:12.009 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:12.022+0000 7f98b727b640 10 monclient(hunting): picked mon.noname-b con 0x7f98b00b1f60 addr v1:172.21.15.84:6790/0 2024-03-29T11:26:12.009 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:12.022+0000 7f98b727b640 10 monclient(hunting): picked mon.noname-c con 0x7f98b0071eb0 addr v1:172.21.15.179:6789/0 2024-03-29T11:26:12.009 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:12.022+0000 7f98b5017640 1 --1- >> v1:172.21.15.84:6790/0 conn(0x7f98b00b1f60 0x7f98b0143600 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.84:6790/0 says I am v1:172.21.15.84:57090/0 (socket says 172.21.15.84:57090) 2024-03-29T11:26:12.009 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:12.022+0000 7f98b5017640 1 -- 172.21.15.84:0/2032468020 learned_addr learned my addr 172.21.15.84:0/2032468020 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-29T11:26:12.010 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:12.022+0000 7f98b727b640 10 monclient(hunting): picked mon.noname-a con 0x7f98b0143d10 addr v1:172.21.15.84:6789/0 2024-03-29T11:26:12.010 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:12.022+0000 7f98b727b640 1 -- 172.21.15.84:0/2032468020 --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98b006ef80 con 0x7f98b0143d10 2024-03-29T11:26:12.010 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:12.022+0000 7f98b727b640 1 -- 172.21.15.84:0/2032468020 --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98b006f260 con 0x7f98b0071eb0 2024-03-29T11:26:12.010 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:12.022+0000 7f98b727b640 1 -- 172.21.15.84:0/2032468020 --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98b00b40c0 con 0x7f98b00b1f60 2024-03-29T11:26:12.010 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:12.022+0000 7f98b727b640 10 monclient(hunting): _renew_subs 2024-03-29T11:26:12.010 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:12.023+0000 7f98b727b640 10 monclient(hunting): authenticate will time out at 1233.317125s 2024-03-29T11:26:12.010 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:12.023+0000 7f98a7fff640 1 -- 172.21.15.84:0/2032468020 <== mon.1 v1:172.21.15.179:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 798002805 0 0) 0x7f98b006f260 con 0x7f98b0071eb0 2024-03-29T11:26:12.010 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:12.023+0000 7f98a7fff640 10 monclient(hunting): _negotiate 2024-03-29T11:26:12.010 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:12.023+0000 7f98a7fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:26:12.010 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:12.023+0000 7f98a7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:26:12.010 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:12.023+0000 7f98a7fff640 10 monclient(hunting): my global_id is 4435 2024-03-29T11:26:12.010 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:12.023+0000 7f98a7fff640 1 -- 172.21.15.84:0/2032468020 --> v1:172.21.15.179:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f989c003520 con 0x7f98b0071eb0 2024-03-29T11:26:12.011 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:12.023+0000 7f98a7fff640 1 -- 172.21.15.84:0/2032468020 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 601814466 0 0) 0x7f98b00b40c0 con 0x7f98b00b1f60 2024-03-29T11:26:12.011 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:12.023+0000 7f98a7fff640 10 monclient(hunting): _negotiate 2024-03-29T11:26:12.011 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:12.023+0000 7f98a7fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:26:12.011 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:12.023+0000 7f98a7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:26:12.011 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:12.023+0000 7f98a7fff640 10 monclient(hunting): my global_id is 4487 2024-03-29T11:26:12.011 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:12.023+0000 7f98a7fff640 1 -- 172.21.15.84:0/2032468020 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f98b006f260 con 0x7f98b00b1f60 2024-03-29T11:26:12.011 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:12.023+0000 7f98a7fff640 1 -- 172.21.15.84:0/2032468020 <== mon.0 v1:172.21.15.84:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2837448239 0 0) 0x7f98b006ef80 con 0x7f98b0143d10 2024-03-29T11:26:12.011 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:12.023+0000 7f98a7fff640 10 monclient(hunting): _negotiate 2024-03-29T11:26:12.011 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:12.023+0000 7f98a7fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:26:12.011 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:12.023+0000 7f98a7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:26:12.011 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:12.023+0000 7f98a7fff640 10 monclient(hunting): my global_id is 4473 2024-03-29T11:26:12.011 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:12.023+0000 7f98a7fff640 1 -- 172.21.15.84:0/2032468020 --> v1:172.21.15.84:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f98b00b40c0 con 0x7f98b0143d10 2024-03-29T11:26:12.011 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:12.023+0000 7f98a7fff640 1 -- 172.21.15.84:0/2032468020 <== mon.2 v1:172.21.15.84:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2002252035 0 0) 0x7f98b006f260 con 0x7f98b00b1f60 2024-03-29T11:26:12.011 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:12.023+0000 7f98a7fff640 1 -- 172.21.15.84:0/2032468020 >> v1:172.21.15.179:6789/0 conn(0x7f98b0071eb0 legacy=0x7f98b0146470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:12.011 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:12.023+0000 7f98a7fff640 1 -- 172.21.15.84:0/2032468020 >> v1:172.21.15.84:6789/0 conn(0x7f98b0143d10 legacy=0x7f98b0146b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:12.012 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:12.023+0000 7f98a7fff640 10 monclient: _finish_hunting 0 2024-03-29T11:26:12.012 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:12.023+0000 7f98a7fff640 1 monclient: found mon.noname-b 2024-03-29T11:26:12.012 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:12.023+0000 7f98a7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:26:12.012 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:12.023+0000 7f98a7fff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.84:6790/0 2024-03-29T11:26:12.012 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:12.023+0000 7f98a7fff640 1 -- 172.21.15.84:0/2032468020 --> v1:172.21.15.84:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98b0152d40 con 0x7f98b00b1f60 2024-03-29T11:26:12.012 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:12.023+0000 7f98a7fff640 10 monclient: _finish_auth 0 2024-03-29T11:26:12.012 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:12.023+0000 7f98a7fff640 10 monclient: _check_auth_tickets 2024-03-29T11:26:12.012 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:12.023+0000 7f98a7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:26:12.012 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:12.023+0000 7f98a7fff640 1 -- 172.21.15.84:0/2032468020 <== mon.2 v1:172.21.15.84:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4236868694 0 0) 0x7f98a80019f0 con 0x7f98b00b1f60 2024-03-29T11:26:12.012 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:12.023+0000 7f98a7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:12.012 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:12.023+0000 7f98a7fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:26:12.012 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:12.023+0000 7f98a7fff640 10 monclient: dump: 2024-03-29T11:26:12.012 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T11:26:12.013 INFO:teuthology.misc.health.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:26:12.013 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T11:18:53.864025+0000 2024-03-29T11:26:12.013 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T11:18:53.864025+0000 2024-03-29T11:26:12.013 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:12.013 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:26:12.013 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T11:26:12.013 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T11:26:12.013 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T11:26:12.013 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:26:12.013 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:12.023+0000 7f98b727b640 5 monclient: authenticate success, global_id 4487 2024-03-29T11:26:12.013 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:12.023+0000 7f98b727b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:26:12.013 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:12.024+0000 7f98a7fff640 1 -- 172.21.15.84:0/2032468020 <== mon.2 v1:172.21.15.84:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f98a8001b50 con 0x7f98b00b1f60 2024-03-29T11:26:12.013 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:12.024+0000 7f98a7fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:26:12.013 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:12.024+0000 7f98a7fff640 1 -- 172.21.15.84:0/2032468020 <== mon.2 v1:172.21.15.84:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4236868694 0 0) 0x7f98a8001f60 con 0x7f98b00b1f60 2024-03-29T11:26:12.013 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:12.024+0000 7f98a7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:12.014 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:12.024+0000 7f98a7fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:26:12.014 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:12.024+0000 7f98a7fff640 10 monclient: dump: 2024-03-29T11:26:12.014 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T11:26:12.014 INFO:teuthology.misc.health.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:26:12.014 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T11:18:53.864025+0000 2024-03-29T11:26:12.014 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T11:18:53.864025+0000 2024-03-29T11:26:12.014 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:12.014 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:26:12.014 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T11:26:12.014 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T11:26:12.014 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T11:26:12.014 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:26:12.014 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:12.024+0000 7f98b727b640 10 monclient: get_monmap_and_config success 2024-03-29T11:26:12.014 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:12.024+0000 7f98b727b640 4 set_mon_vals no callback set 2024-03-29T11:26:12.019 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:12.025+0000 7f98b727b640 10 monclient: shutdown 2024-03-29T11:26:12.019 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:12.025+0000 7f98b727b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:26:12.019 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:12.025+0000 7f98b727b640 1 -- 172.21.15.84:0/2032468020 >> v1:172.21.15.84:6790/0 conn(0x7f98b00b1f60 legacy=0x7f98b0143600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:12.019 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:12.025+0000 7f98b727b640 1 -- 172.21.15.84:0/2032468020 shutdown_connections 2024-03-29T11:26:12.019 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:12.025+0000 7f98b727b640 1 -- 172.21.15.84:0/2032468020 >> 172.21.15.84:0/2032468020 conn(0x7f98b0066a00 msgr2=0x7f98b0066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:26:12.019 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:12.025+0000 7f98b727b640 1 -- 172.21.15.84:0/2032468020 shutdown_connections 2024-03-29T11:26:12.019 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:12.025+0000 7f98b727b640 1 -- 172.21.15.84:0/2032468020 wait complete. 2024-03-29T11:26:12.019 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:12.026+0000 7f98b727b640 10 monclient: build_initial_monmap 2024-03-29T11:26:12.019 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:12.026+0000 7f98b727b640 10 monclient: monmap: 2024-03-29T11:26:12.019 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-29T11:26:12.019 INFO:teuthology.misc.health.smithi084.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:26:12.020 INFO:teuthology.misc.health.smithi084.stderr:last_changed 0.000000 2024-03-29T11:26:12.020 INFO:teuthology.misc.health.smithi084.stderr:created 0.000000 2024-03-29T11:26:12.020 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-29T11:26:12.020 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:26:12.020 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-29T11:26:12.020 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.noname-b 2024-03-29T11:26:12.020 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-c 2024-03-29T11:26:12.020 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:26:12.020 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:12.027+0000 7f98b727b640 1 librados: starting msgr at 2024-03-29T11:26:12.020 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:12.027+0000 7f98b727b640 1 librados: starting objecter 2024-03-29T11:26:12.020 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:12.027+0000 7f98b727b640 1 Processor -- start 2024-03-29T11:26:12.020 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:12.027+0000 7f98b727b640 1 -- start start 2024-03-29T11:26:12.020 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:12.027+0000 7f98b727b640 1 librados: setting wanted keys 2024-03-29T11:26:12.020 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:12.027+0000 7f98b727b640 1 librados: calling monclient init 2024-03-29T11:26:12.021 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:12.027+0000 7f98b727b640 10 monclient: init 2024-03-29T11:26:12.021 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:12.028+0000 7f98b727b640 10 monclient: _reopen_session rank -1 2024-03-29T11:26:12.021 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:12.028+0000 7f98b727b640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:26:12.021 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:12.028+0000 7f98b727b640 10 monclient(hunting): picked mon.noname-c con 0x7f98b00b1f60 addr v1:172.21.15.84:6790/0 2024-03-29T11:26:12.021 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:12.028+0000 7f98b727b640 10 monclient(hunting): picked mon.noname-a con 0x7f98b0143d10 addr v1:172.21.15.84:6789/0 2024-03-29T11:26:12.021 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:12.028+0000 7f98b727b640 10 monclient(hunting): picked mon.noname-b con 0x7f98b013dc90 addr v1:172.21.15.179:6789/0 2024-03-29T11:26:12.021 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:12.028+0000 7f98b5017640 1 --1- >> v1:172.21.15.84:6790/0 conn(0x7f98b00b1f60 0x7f98b01defc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.84:6790/0 says I am v1:172.21.15.84:57396/0 (socket says 172.21.15.84:57396) 2024-03-29T11:26:12.021 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:12.028+0000 7f98b5017640 1 -- 172.21.15.84:0/327433144 learned_addr learned my addr 172.21.15.84:0/327433144 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-29T11:26:12.021 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:12.028+0000 7f98b727b640 1 -- 172.21.15.84:0/327433144 --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98b01de850 con 0x7f98b0143d10 2024-03-29T11:26:12.021 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:12.028+0000 7f98b727b640 1 -- 172.21.15.84:0/327433144 --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98b01df6d0 con 0x7f98b013dc90 2024-03-29T11:26:12.021 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:12.028+0000 7f98a5ffb640 10 client.?.objecter ms_handle_connect 0x7f98b00b1f60 2024-03-29T11:26:12.021 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:12.028+0000 7f98a5ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:26:12.022 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:12.028+0000 7f98a5ffb640 10 client.?.objecter ms_handle_connect 0x7f98b0143d10 2024-03-29T11:26:12.022 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:12.028+0000 7f98a5ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:26:12.022 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:12.028+0000 7f98b727b640 1 -- 172.21.15.84:0/327433144 --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98b01deaa0 con 0x7f98b00b1f60 2024-03-29T11:26:12.022 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:12.028+0000 7f98b727b640 10 monclient(hunting): _renew_subs 2024-03-29T11:26:12.022 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:12.028+0000 7f98b727b640 10 monclient(hunting): authenticate will time out at 1233.322814s 2024-03-29T11:26:12.022 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:12.028+0000 7f98a5ffb640 10 client.?.objecter ms_handle_connect 0x7f98b013dc90 2024-03-29T11:26:12.022 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:12.028+0000 7f98a5ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:26:12.022 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:12.030+0000 7f98a5ffb640 1 -- 172.21.15.84:0/327433144 <== mon.0 v1:172.21.15.84:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2074693050 0 0) 0x7f98b01de850 con 0x7f98b0143d10 2024-03-29T11:26:12.022 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:12.030+0000 7f98a5ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:26:12.022 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:12.030+0000 7f98a5ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:26:12.022 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:12.030+0000 7f98a5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:26:12.022 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:12.030+0000 7f98a5ffb640 10 monclient(hunting): my global_id is 4476 2024-03-29T11:26:12.022 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:12.030+0000 7f98a5ffb640 1 -- 172.21.15.84:0/327433144 --> v1:172.21.15.84:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9894004eb0 con 0x7f98b0143d10 2024-03-29T11:26:12.022 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:12.031+0000 7f98a5ffb640 1 -- 172.21.15.84:0/327433144 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1120106127 0 0) 0x7f98b01deaa0 con 0x7f98b00b1f60 2024-03-29T11:26:12.023 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:12.031+0000 7f98a5ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:26:12.023 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:12.031+0000 7f98a5ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:26:12.023 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:12.031+0000 7f98a5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:26:12.023 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:12.031+0000 7f98a5ffb640 10 monclient(hunting): my global_id is 4490 2024-03-29T11:26:12.023 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:12.031+0000 7f98a5ffb640 1 -- 172.21.15.84:0/327433144 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f98b01de850 con 0x7f98b00b1f60 2024-03-29T11:26:12.023 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:12.031+0000 7f98a5ffb640 1 -- 172.21.15.84:0/327433144 <== mon.0 v1:172.21.15.84:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1170185625 0 0) 0x7f9894004eb0 con 0x7f98b0143d10 2024-03-29T11:26:12.023 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:12.031+0000 7f98a5ffb640 1 -- 172.21.15.84:0/327433144 --> v1:172.21.15.84:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f98b01deaa0 con 0x7f98b0143d10 2024-03-29T11:26:12.023 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:12.031+0000 7f98a5ffb640 1 -- 172.21.15.84:0/327433144 <== mon.0 v1:172.21.15.84:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4236868694 0 0) 0x7f98ac001950 con 0x7f98b0143d10 2024-03-29T11:26:12.023 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:12.031+0000 7f98a5ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:12.023 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:12.031+0000 7f98a5ffb640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:26:12.023 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:12.031+0000 7f98a5ffb640 10 monclient(hunting): dump: 2024-03-29T11:26:12.023 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T11:26:12.023 INFO:teuthology.misc.health.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:26:12.023 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T11:18:53.864025+0000 2024-03-29T11:26:12.024 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T11:18:53.864025+0000 2024-03-29T11:26:12.024 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:12.024 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:26:12.024 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T11:26:12.024 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T11:26:12.024 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T11:26:12.024 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:26:12.024 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:12.031+0000 7f98a5ffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:26:12.024 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:12.031+0000 7f98a5ffb640 1 -- 172.21.15.84:0/327433144 <== mon.0 v1:172.21.15.84:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 955137767 0 0) 0x7f98b01deaa0 con 0x7f98b0143d10 2024-03-29T11:26:12.024 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:12.031+0000 7f98a5ffb640 1 -- 172.21.15.84:0/327433144 >> v1:172.21.15.84:6790/0 conn(0x7f98b00b1f60 legacy=0x7f98b01defc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:12.024 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:12.031+0000 7f98a5ffb640 1 -- 172.21.15.84:0/327433144 >> v1:172.21.15.179:6789/0 conn(0x7f98b013dc90 legacy=0x7f98b01db6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:12.024 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:12.031+0000 7f98a5ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:26:12.024 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:12.031+0000 7f98a5ffb640 1 monclient: found mon.a 2024-03-29T11:26:12.024 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:12.031+0000 7f98a5ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:26:12.025 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:12.031+0000 7f98a5ffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.84:6789/0 2024-03-29T11:26:12.025 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:12.031+0000 7f98a5ffb640 1 -- 172.21.15.84:0/327433144 --> v1:172.21.15.84:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98b01db350 con 0x7f98b0143d10 2024-03-29T11:26:12.025 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:12.031+0000 7f98a5ffb640 10 monclient: _finish_auth 0 2024-03-29T11:26:12.025 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:12.031+0000 7f98a5ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:26:12.025 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:12.031+0000 7f98a5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:26:12.025 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:12.031+0000 7f98b727b640 5 monclient: authenticate success, global_id 4476 2024-03-29T11:26:12.025 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:12.031+0000 7f98b727b640 10 monclient: _renew_subs 2024-03-29T11:26:12.025 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:12.031+0000 7f98b727b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.84:6789/0 2024-03-29T11:26:12.025 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:12.031+0000 7f98b727b640 1 -- 172.21.15.84:0/327433144 --> v1:172.21.15.84:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f98b013c230 con 0x7f98b0143d10 2024-03-29T11:26:12.025 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:12.032+0000 7f98b727b640 10 client.4476.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:26:12.025 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:12.032+0000 7f98b727b640 10 monclient: _renew_subs 2024-03-29T11:26:12.025 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:12.032+0000 7f98b727b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.84:6789/0 2024-03-29T11:26:12.025 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:12.032+0000 7f98b727b640 1 -- 172.21.15.84:0/327433144 --> v1:172.21.15.84:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f98b01dbdc0 con 0x7f98b0143d10 2024-03-29T11:26:12.025 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:12.032+0000 7f98b727b640 1 librados: init done 2024-03-29T11:26:12.026 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:12.032+0000 7f98a5ffb640 1 -- 172.21.15.84:0/327433144 <== mon.0 v1:172.21.15.84:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f98ac003610 con 0x7f98b0143d10 2024-03-29T11:26:12.026 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:12.032+0000 7f98a5ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:26:12.026 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:12.032+0000 7f98a5ffb640 1 -- 172.21.15.84:0/327433144 <== mon.0 v1:172.21.15.84:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4236868694 0 0) 0x7f98ac001950 con 0x7f98b0143d10 2024-03-29T11:26:12.026 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:12.032+0000 7f98a77fe640 4 set_mon_vals no callback set 2024-03-29T11:26:12.026 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:12.032+0000 7f98a5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:12.026 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:12.032+0000 7f98a5ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:26:12.026 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:12.032+0000 7f98a5ffb640 10 monclient: dump: 2024-03-29T11:26:12.026 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T11:26:12.026 INFO:teuthology.misc.health.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:26:12.026 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T11:18:53.864025+0000 2024-03-29T11:26:12.026 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T11:18:53.864025+0000 2024-03-29T11:26:12.026 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:12.026 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:26:12.026 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T11:26:12.026 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T11:26:12.027 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T11:26:12.027 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:26:12.028 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:12.034+0000 7f98b727b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:26:12.028 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:12.034+0000 7f98b727b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:26:12.028 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:12.034+0000 7f98b727b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.84:6789/0 2024-03-29T11:26:12.028 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:12.034+0000 7f98b727b640 1 -- 172.21.15.84:0/327433144 --> v1:172.21.15.84:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9890005080 con 0x7f98b0143d10 2024-03-29T11:26:12.028 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:12.034+0000 7f98a5ffb640 1 -- 172.21.15.84:0/327433144 <== mon.0 v1:172.21.15.84:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4033942909 0 0) 0x7f98ac003880 con 0x7f98b0143d10 2024-03-29T11:26:12.028 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:12.035+0000 7f98a5ffb640 1 -- 172.21.15.84:0/327433144 <== mon.0 v1:172.21.15.84:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2808139587 0 0) 0x7f98ac0832b0 con 0x7f98b0143d10 2024-03-29T11:26:12.028 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:12.035+0000 7f98a5ffb640 10 client.4476.objecter ms_dispatch 0x7f98b0000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:26:12.028 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:12.035+0000 7f98a5ffb640 3 client.4476.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:26:12.028 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:12.035+0000 7f98a5ffb640 3 client.4476.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:26:12.029 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:12.035+0000 7f98a5ffb640 20 client.4476.objecter dump_active .. 0 homeless 2024-03-29T11:26:12.029 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:12.035+0000 7f98b5017640 10 monclient: get_auth_request con 0x7f989406e090 auth_method 0 2024-03-29T11:26:12.029 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:12.035+0000 7f98a5ffb640 10 client.4476.objecter ms_handle_connect 0x7f989406e090 2024-03-29T11:26:12.032 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:12.046+0000 7f98a5ffb640 1 -- 172.21.15.84:0/327433144 <== mon.0 v1:172.21.15.84:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f98ac04ecd0 con 0x7f98b0143d10 2024-03-29T11:26:12.032 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:12.046+0000 7f98a5ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:26:12.032 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:12.046+0000 7f98a5ffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:26:12.386 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:12.400+0000 7f98b727b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:26:12.386 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:12.400+0000 7f98b727b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:26:12.386 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:12.400+0000 7f98b727b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.84:6789/0 2024-03-29T11:26:12.386 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:12.400+0000 7f98b727b640 1 -- 172.21.15.84:0/327433144 --> v1:172.21.15.84:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9890005330 con 0x7f98b0143d10 2024-03-29T11:26:12.387 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:12.401+0000 7f98a5ffb640 1 -- 172.21.15.84:0/327433144 <== mon.0 v1:172.21.15.84:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4209477549) 0x7f98ac050c00 con 0x7f98b0143d10 2024-03-29T11:26:12.387 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:12.401+0000 7f98a5ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:26:12.387 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:12.401+0000 7f98a5ffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:26:12.387 INFO:teuthology.misc.health.smithi084.stdout: 2024-03-29T11:26:12.387 INFO:teuthology.misc.health.smithi084.stdout:{"epoch":10,"fsid":"2c3e002f-9eb0-4860-853b-4ef1ef562213","created":"2024-03-29T11:19:32.029169+0000","modified":"2024-03-29T11:19:34.200299+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:34.200299+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"5c39e276-2034-4716-bbc0-57a634eefa92","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"cd580e99-109f-4bf1-88c0-a5987bc8b9ed","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"11e60a5a-051d-453f-a190-66387cb35592","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"2c329fbe-bbc4-4030-bf87-d711cad04679","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f4213d4a-bd36-4433-91ae-d7e2b02481d3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"f0cf44ea-6055-48af-805b-aed858102ff9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"d895eaa0-cc0b-4ee4-9b5a-2dfcd89a60b7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"9bc5a7bc-8f52-4db9-96d6-de5e68d52eaa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:26:12.391 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:12.404+0000 7f98b727b640 10 librados: watch_flush enter 2024-03-29T11:26:12.391 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:12.404+0000 7f98b727b640 10 librados: watch_flush exit 2024-03-29T11:26:12.391 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:12.405+0000 7f98b727b640 20 client.4476.objecter shutdown clearing up homeless session... 2024-03-29T11:26:12.391 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:12.405+0000 7f98b727b640 10 client.4476.objecter successfully canceled tick 2024-03-29T11:26:12.391 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:12.405+0000 7f98b727b640 1 -- 172.21.15.84:0/327433144 >> v1:172.21.15.179:6816/2981134980 conn(0x7f989406e090 legacy=0x7f9894070550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:12.391 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:12.405+0000 7f98b727b640 10 monclient: shutdown 2024-03-29T11:26:12.391 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:12.405+0000 7f98b727b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:26:12.391 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:12.405+0000 7f98b727b640 1 -- 172.21.15.84:0/327433144 >> v1:172.21.15.84:6789/0 conn(0x7f98b0143d10 legacy=0x7f98b013d580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:12.391 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:12.405+0000 7f98b727b640 1 -- 172.21.15.84:0/327433144 shutdown_connections 2024-03-29T11:26:12.391 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:12.405+0000 7f98b727b640 1 -- 172.21.15.84:0/327433144 >> 172.21.15.84:0/327433144 conn(0x7f98b0066a00 msgr2=0x7f98b0141bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:26:12.392 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:12.405+0000 7f98b727b640 1 -- 172.21.15.84:0/327433144 shutdown_connections 2024-03-29T11:26:12.392 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:12.405+0000 7f98b727b640 1 -- 172.21.15.84:0/327433144 wait complete. 2024-03-29T11:26:12.392 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:12.405+0000 7f98b727b640 1 librados: shutdown 2024-03-29T11:26:12.407 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:26:19.213 DEBUG:teuthology.orchestra.run.smithi084:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:26:19.344 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:19.357+0000 7f449a182640 10 monclient: get_monmap_and_config 2024-03-29T11:26:19.345 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:19.357+0000 7f449a182640 10 monclient: build_initial_monmap 2024-03-29T11:26:19.345 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:19.357+0000 7f449a182640 10 monclient: monmap: 2024-03-29T11:26:19.345 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-29T11:26:19.345 INFO:teuthology.misc.health.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:26:19.345 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T11:26:19.359822+0000 2024-03-29T11:26:19.345 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T11:26:19.359822+0000 2024-03-29T11:26:19.345 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-29T11:26:19.345 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:26:19.345 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-29T11:26:19.345 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.noname-c 2024-03-29T11:26:19.345 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-b 2024-03-29T11:26:19.345 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:26:19.346 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:19.359+0000 7f449a182640 1 Processor -- start 2024-03-29T11:26:19.346 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:19.359+0000 7f449a182640 1 -- start start 2024-03-29T11:26:19.346 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:19.360+0000 7f449a182640 10 monclient: init 2024-03-29T11:26:19.348 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:19.360+0000 7f449a182640 10 monclient: _reopen_session rank -1 2024-03-29T11:26:19.348 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:19.360+0000 7f449a182640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:26:19.348 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:19.360+0000 7f449a182640 10 monclient(hunting): picked mon.noname-c con 0x7f44940b0860 addr v1:172.21.15.179:6789/0 2024-03-29T11:26:19.348 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:19.360+0000 7f449a182640 10 monclient(hunting): picked mon.noname-a con 0x7f4494072450 addr v1:172.21.15.84:6789/0 2024-03-29T11:26:19.348 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:19.360+0000 7f449a182640 10 monclient(hunting): picked mon.noname-b con 0x7f449413ee40 addr v1:172.21.15.84:6790/0 2024-03-29T11:26:19.348 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:19.360+0000 7f449a182640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f44940ad840 con 0x7f4494072450 2024-03-29T11:26:19.348 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:19.360+0000 7f449a182640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f449413c830 con 0x7f44940b0860 2024-03-29T11:26:19.348 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:19.360+0000 7f449a182640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f449413c9c0 con 0x7f449413ee40 2024-03-29T11:26:19.349 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:19.360+0000 7f449a182640 10 monclient(hunting): _renew_subs 2024-03-29T11:26:19.349 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:19.360+0000 7f449a182640 10 monclient(hunting): authenticate will time out at 1240.656055s 2024-03-29T11:26:19.349 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:19.361+0000 7f4493fff640 1 --1- >> v1:172.21.15.84:6790/0 conn(0x7f449413ee40 0x7f449413f260 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.84:6790/0 says I am v1:172.21.15.84:46906/0 (socket says 172.21.15.84:46906) 2024-03-29T11:26:19.349 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:19.361+0000 7f4493fff640 1 -- 172.21.15.84:0/2419156364 learned_addr learned my addr 172.21.15.84:0/2419156364 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-29T11:26:19.349 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:19.361+0000 7f44927fc640 1 -- 172.21.15.84:0/2419156364 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3591602507 0 0) 0x7f449413c9c0 con 0x7f449413ee40 2024-03-29T11:26:19.349 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:19.361+0000 7f44927fc640 10 monclient(hunting): _negotiate 2024-03-29T11:26:19.349 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:19.361+0000 7f44927fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:26:19.349 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:19.361+0000 7f44927fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:26:19.349 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:19.361+0000 7f44927fc640 10 monclient(hunting): my global_id is 4493 2024-03-29T11:26:19.349 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:19.361+0000 7f44927fc640 1 -- 172.21.15.84:0/2419156364 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f447c003520 con 0x7f449413ee40 2024-03-29T11:26:19.349 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:19.361+0000 7f44927fc640 1 -- 172.21.15.84:0/2419156364 <== mon.2 v1:172.21.15.84:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3695001899 0 0) 0x7f447c003520 con 0x7f449413ee40 2024-03-29T11:26:19.349 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:19.361+0000 7f44927fc640 1 -- 172.21.15.84:0/2419156364 >> v1:172.21.15.179:6789/0 conn(0x7f44940b0860 legacy=0x7f44940acfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:26:19.349 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:19.361+0000 7f44927fc640 1 -- 172.21.15.84:0/2419156364 >> v1:172.21.15.84:6789/0 conn(0x7f4494072450 legacy=0x7f449413c120 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:26:19.353 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:19.365+0000 7f44927fc640 10 monclient: _finish_hunting 0 2024-03-29T11:26:19.353 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:19.365+0000 7f44927fc640 1 monclient: found mon.noname-b 2024-03-29T11:26:19.353 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:19.365+0000 7f44927fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:26:19.353 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:19.365+0000 7f44927fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.84:6790/0 2024-03-29T11:26:19.353 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:19.365+0000 7f44927fc640 1 -- 172.21.15.84:0/2419156364 --> v1:172.21.15.84:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f449413f970 con 0x7f449413ee40 2024-03-29T11:26:19.353 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:19.365+0000 7f44927fc640 10 monclient: _finish_auth 0 2024-03-29T11:26:19.353 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:19.365+0000 7f44927fc640 10 monclient: _check_auth_tickets 2024-03-29T11:26:19.353 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:19.365+0000 7f44927fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:26:19.353 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:19.365+0000 7f44927fc640 1 -- 172.21.15.84:0/2419156364 <== mon.2 v1:172.21.15.84:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4236868694 0 0) 0x7f448c001a10 con 0x7f449413ee40 2024-03-29T11:26:19.353 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:19.365+0000 7f44927fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:19.353 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:19.365+0000 7f44927fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:26:19.354 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:19.365+0000 7f44927fc640 10 monclient: dump: 2024-03-29T11:26:19.354 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T11:26:19.354 INFO:teuthology.misc.health.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:26:19.354 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T11:18:53.864025+0000 2024-03-29T11:26:19.354 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T11:18:53.864025+0000 2024-03-29T11:26:19.354 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:19.354 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:26:19.354 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T11:26:19.354 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T11:26:19.354 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T11:26:19.354 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:26:19.354 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:19.365+0000 7f449a182640 5 monclient: authenticate success, global_id 4493 2024-03-29T11:26:19.354 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:19.365+0000 7f449a182640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:26:19.356 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:19.365+0000 7f44927fc640 1 -- 172.21.15.84:0/2419156364 <== mon.2 v1:172.21.15.84:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f448c001c70 con 0x7f449413ee40 2024-03-29T11:26:19.356 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:19.365+0000 7f44927fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:26:19.356 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:19.365+0000 7f449a182640 10 monclient: get_monmap_and_config success 2024-03-29T11:26:19.356 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:19.365+0000 7f449a182640 4 set_mon_vals no callback set 2024-03-29T11:26:19.356 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:19.366+0000 7f44927fc640 1 -- 172.21.15.84:0/2419156364 <== mon.2 v1:172.21.15.84:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4236868694 0 0) 0x7f448c0020c0 con 0x7f449413ee40 2024-03-29T11:26:19.356 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:19.366+0000 7f44927fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:19.356 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:19.366+0000 7f44927fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:26:19.356 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:19.366+0000 7f44927fc640 10 monclient: dump: 2024-03-29T11:26:19.356 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T11:26:19.356 INFO:teuthology.misc.health.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:26:19.357 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T11:18:53.864025+0000 2024-03-29T11:26:19.357 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T11:18:53.864025+0000 2024-03-29T11:26:19.357 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:19.357 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:26:19.357 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T11:26:19.357 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T11:26:19.357 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T11:26:19.357 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:26:19.357 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:19.366+0000 7f449a182640 10 monclient: shutdown 2024-03-29T11:26:19.357 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:19.366+0000 7f449a182640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:26:19.357 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:19.366+0000 7f449a182640 1 -- 172.21.15.84:0/2419156364 >> v1:172.21.15.84:6790/0 conn(0x7f449413ee40 legacy=0x7f449413f260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:19.357 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:19.366+0000 7f449a182640 1 -- 172.21.15.84:0/2419156364 shutdown_connections 2024-03-29T11:26:19.358 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:19.366+0000 7f449a182640 1 -- 172.21.15.84:0/2419156364 >> 172.21.15.84:0/2419156364 conn(0x7f449406b660 msgr2=0x7f449406ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:26:19.358 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:19.367+0000 7f449a182640 1 -- 172.21.15.84:0/2419156364 shutdown_connections 2024-03-29T11:26:19.358 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:19.367+0000 7f449a182640 1 -- 172.21.15.84:0/2419156364 wait complete. 2024-03-29T11:26:19.358 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:19.367+0000 7f449a182640 10 monclient: build_initial_monmap 2024-03-29T11:26:19.358 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:19.367+0000 7f449a182640 10 monclient: monmap: 2024-03-29T11:26:19.358 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-29T11:26:19.358 INFO:teuthology.misc.health.smithi084.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:26:19.358 INFO:teuthology.misc.health.smithi084.stderr:last_changed 0.000000 2024-03-29T11:26:19.358 INFO:teuthology.misc.health.smithi084.stderr:created 0.000000 2024-03-29T11:26:19.358 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-29T11:26:19.358 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:26:19.359 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-29T11:26:19.359 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.noname-b 2024-03-29T11:26:19.359 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-c 2024-03-29T11:26:19.359 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:26:19.359 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:19.368+0000 7f449a182640 1 librados: starting msgr at 2024-03-29T11:26:19.359 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:19.368+0000 7f449a182640 1 librados: starting objecter 2024-03-29T11:26:19.359 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:19.368+0000 7f449a182640 1 Processor -- start 2024-03-29T11:26:19.359 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:19.368+0000 7f449a182640 1 -- start start 2024-03-29T11:26:19.359 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:19.368+0000 7f449a182640 1 librados: setting wanted keys 2024-03-29T11:26:19.359 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:19.368+0000 7f449a182640 1 librados: calling monclient init 2024-03-29T11:26:19.359 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:19.368+0000 7f449a182640 10 monclient: init 2024-03-29T11:26:19.359 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:19.369+0000 7f449a182640 10 monclient: _reopen_session rank -1 2024-03-29T11:26:19.359 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:19.369+0000 7f449a182640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:26:19.366 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:19.370+0000 7f449a182640 10 monclient(hunting): picked mon.noname-a con 0x7f44940b0860 addr v1:172.21.15.84:6789/0 2024-03-29T11:26:19.366 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:19.370+0000 7f449a182640 10 monclient(hunting): picked mon.noname-c con 0x7f449413ee40 addr v1:172.21.15.84:6790/0 2024-03-29T11:26:19.366 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:19.370+0000 7f449a182640 10 monclient(hunting): picked mon.noname-b con 0x7f44940b2020 addr v1:172.21.15.179:6789/0 2024-03-29T11:26:19.366 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:19.370+0000 7f449a182640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f44940b4170 con 0x7f44940b0860 2024-03-29T11:26:19.366 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:19.371+0000 7f449a182640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f449413e9d0 con 0x7f44940b2020 2024-03-29T11:26:19.366 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:19.371+0000 7f449a182640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f449413e540 con 0x7f449413ee40 2024-03-29T11:26:19.366 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:19.371+0000 7f449a182640 10 monclient(hunting): _renew_subs 2024-03-29T11:26:19.366 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:19.371+0000 7f449a182640 10 monclient(hunting): authenticate will time out at 1240.666163s 2024-03-29T11:26:19.366 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:19.371+0000 7f4492ffd640 1 --1- >> v1:172.21.15.84:6790/0 conn(0x7f449413ee40 0x7f449406de90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.84:6790/0 says I am v1:172.21.15.84:47988/0 (socket says 172.21.15.84:47988) 2024-03-29T11:26:19.366 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:19.371+0000 7f4492ffd640 1 -- 172.21.15.84:0/1703214844 learned_addr learned my addr 172.21.15.84:0/1703214844 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-29T11:26:19.367 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:19.371+0000 7f447bfff640 10 client.?.objecter ms_handle_connect 0x7f449413ee40 2024-03-29T11:26:19.367 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:19.371+0000 7f447bfff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:26:19.367 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:19.371+0000 7f447bfff640 10 client.?.objecter ms_handle_connect 0x7f44940b2020 2024-03-29T11:26:19.367 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:19.371+0000 7f447bfff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:26:19.367 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:19.371+0000 7f447bfff640 1 -- 172.21.15.84:0/1703214844 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 963578583 0 0) 0x7f449413e540 con 0x7f449413ee40 2024-03-29T11:26:19.367 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:19.371+0000 7f447bfff640 10 monclient(hunting): _negotiate 2024-03-29T11:26:19.367 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:19.371+0000 7f447bfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:26:19.367 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:19.371+0000 7f447bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:26:19.367 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:19.371+0000 7f447bfff640 10 monclient(hunting): my global_id is 4496 2024-03-29T11:26:19.367 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:19.371+0000 7f447bfff640 1 -- 172.21.15.84:0/1703214844 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4480004eb0 con 0x7f449413ee40 2024-03-29T11:26:19.367 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:19.371+0000 7f447bfff640 1 -- 172.21.15.84:0/1703214844 <== mon.2 v1:172.21.15.84:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2487913481 0 0) 0x7f4480004eb0 con 0x7f449413ee40 2024-03-29T11:26:19.367 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:19.372+0000 7f447bfff640 1 -- 172.21.15.84:0/1703214844 --> v1:172.21.15.84:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f449413e540 con 0x7f449413ee40 2024-03-29T11:26:19.367 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:19.372+0000 7f447bfff640 1 -- 172.21.15.84:0/1703214844 <== mon.2 v1:172.21.15.84:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4236868694 0 0) 0x7f44840018f0 con 0x7f449413ee40 2024-03-29T11:26:19.368 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:19.372+0000 7f447bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:19.368 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:19.372+0000 7f447bfff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:26:19.368 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:19.372+0000 7f447bfff640 10 monclient(hunting): dump: 2024-03-29T11:26:19.368 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T11:26:19.368 INFO:teuthology.misc.health.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:26:19.368 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T11:18:53.864025+0000 2024-03-29T11:26:19.368 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T11:18:53.864025+0000 2024-03-29T11:26:19.368 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:19.368 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:26:19.368 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T11:26:19.368 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T11:26:19.368 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T11:26:19.368 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:26:19.368 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:19.372+0000 7f447bfff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:26:19.369 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:19.372+0000 7f447bfff640 1 -- 172.21.15.84:0/1703214844 <== mon.2 v1:172.21.15.84:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1814723708 0 0) 0x7f449413e540 con 0x7f449413ee40 2024-03-29T11:26:19.369 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:19.372+0000 7f447bfff640 1 -- 172.21.15.84:0/1703214844 >> v1:172.21.15.179:6789/0 conn(0x7f44940b2020 legacy=0x7f449406e5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:19.369 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:19.372+0000 7f447bfff640 1 -- 172.21.15.84:0/1703214844 >> v1:172.21.15.84:6789/0 conn(0x7f44940b0860 legacy=0x7f44940b2d10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:26:19.369 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:19.373+0000 7f447bfff640 10 monclient: _finish_hunting 0 2024-03-29T11:26:19.369 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:19.373+0000 7f447bfff640 1 monclient: found mon.c 2024-03-29T11:26:19.369 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:19.373+0000 7f447bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:26:19.369 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:19.373+0000 7f447bfff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-29T11:26:19.369 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:19.373+0000 7f447bfff640 1 -- 172.21.15.84:0/1703214844 --> v1:172.21.15.84:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f449406da70 con 0x7f449413ee40 2024-03-29T11:26:19.369 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:19.373+0000 7f447bfff640 10 monclient: _finish_auth 0 2024-03-29T11:26:19.369 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:19.373+0000 7f447bfff640 10 monclient: _check_auth_tickets 2024-03-29T11:26:19.369 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:19.373+0000 7f447bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:26:19.369 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:19.373+0000 7f449a182640 5 monclient: authenticate success, global_id 4496 2024-03-29T11:26:19.369 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:19.373+0000 7f449a182640 10 monclient: _renew_subs 2024-03-29T11:26:19.369 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:19.373+0000 7f449a182640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-29T11:26:19.370 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:19.373+0000 7f449a182640 1 -- 172.21.15.84:0/1703214844 --> v1:172.21.15.84:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f44941e2cb0 con 0x7f449413ee40 2024-03-29T11:26:19.370 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:19.373+0000 7f449a182640 10 client.4496.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:26:19.370 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:19.373+0000 7f449a182640 10 monclient: _renew_subs 2024-03-29T11:26:19.370 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:19.373+0000 7f449a182640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-29T11:26:19.370 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:19.373+0000 7f449a182640 1 -- 172.21.15.84:0/1703214844 --> v1:172.21.15.84:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f44941e3080 con 0x7f449413ee40 2024-03-29T11:26:19.370 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:19.373+0000 7f449a182640 1 librados: init done 2024-03-29T11:26:19.370 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:19.373+0000 7f447bfff640 1 -- 172.21.15.84:0/1703214844 <== mon.2 v1:172.21.15.84:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4484003610 con 0x7f449413ee40 2024-03-29T11:26:19.370 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:19.373+0000 7f447bfff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:26:19.370 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:19.373+0000 7f4491ffb640 4 set_mon_vals no callback set 2024-03-29T11:26:19.370 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:19.374+0000 7f449a182640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:26:19.370 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:19.374+0000 7f447bfff640 1 -- 172.21.15.84:0/1703214844 <== mon.2 v1:172.21.15.84:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4236868694 0 0) 0x7f44840039e0 con 0x7f449413ee40 2024-03-29T11:26:19.370 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:19.374+0000 7f447bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:19.370 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:19.374+0000 7f447bfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:26:19.370 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:19.374+0000 7f447bfff640 10 monclient: dump: 2024-03-29T11:26:19.371 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T11:26:19.371 INFO:teuthology.misc.health.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:26:19.371 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T11:18:53.864025+0000 2024-03-29T11:26:19.371 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T11:18:53.864025+0000 2024-03-29T11:26:19.371 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:19.371 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:26:19.371 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T11:26:19.371 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T11:26:19.371 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T11:26:19.371 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:26:19.371 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:19.374+0000 7f449a182640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:26:19.371 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:19.374+0000 7f449a182640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-29T11:26:19.371 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:19.374+0000 7f449a182640 1 -- 172.21.15.84:0/1703214844 --> v1:172.21.15.84:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4474005080 con 0x7f449413ee40 2024-03-29T11:26:19.371 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:19.375+0000 7f447bfff640 1 -- 172.21.15.84:0/1703214844 <== mon.2 v1:172.21.15.84:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4033942909 0 0) 0x7f4484003c00 con 0x7f449413ee40 2024-03-29T11:26:19.372 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:19.375+0000 7f447bfff640 1 -- 172.21.15.84:0/1703214844 <== mon.2 v1:172.21.15.84:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2808139587 0 0) 0x7f44840849e0 con 0x7f449413ee40 2024-03-29T11:26:19.372 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:19.375+0000 7f447bfff640 10 client.4496.objecter ms_dispatch 0x7f4494000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:26:19.372 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:19.375+0000 7f447bfff640 3 client.4496.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:26:19.372 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:19.375+0000 7f447bfff640 3 client.4496.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:26:19.372 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:19.376+0000 7f44937fe640 10 monclient: get_auth_request con 0x7f448006dd50 auth_method 0 2024-03-29T11:26:19.372 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:19.376+0000 7f447bfff640 20 client.4496.objecter dump_active .. 0 homeless 2024-03-29T11:26:19.372 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:19.376+0000 7f447bfff640 10 client.4496.objecter ms_handle_connect 0x7f448006dd50 2024-03-29T11:26:19.372 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:19.382+0000 7f447bfff640 1 -- 172.21.15.84:0/1703214844 <== mon.2 v1:172.21.15.84:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f44840502c0 con 0x7f449413ee40 2024-03-29T11:26:19.372 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:19.382+0000 7f447bfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:26:19.373 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:19.382+0000 7f447bfff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:26:19.677 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:19.690+0000 7f449a182640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:26:19.677 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:19.690+0000 7f449a182640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:26:19.677 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:19.690+0000 7f449a182640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-29T11:26:19.677 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:19.690+0000 7f449a182640 1 -- 172.21.15.84:0/1703214844 --> v1:172.21.15.84:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4474005330 con 0x7f449413ee40 2024-03-29T11:26:19.678 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:19.691+0000 7f447bfff640 1 -- 172.21.15.84:0/1703214844 <== mon.2 v1:172.21.15.84:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4209477549) 0x7f44840521f0 con 0x7f449413ee40 2024-03-29T11:26:19.678 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:19.691+0000 7f447bfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:26:19.678 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:19.691+0000 7f447bfff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:26:19.678 INFO:teuthology.misc.health.smithi084.stdout: 2024-03-29T11:26:19.678 INFO:teuthology.misc.health.smithi084.stdout:{"epoch":10,"fsid":"2c3e002f-9eb0-4860-853b-4ef1ef562213","created":"2024-03-29T11:19:32.029169+0000","modified":"2024-03-29T11:19:34.200299+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:34.200299+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"5c39e276-2034-4716-bbc0-57a634eefa92","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"cd580e99-109f-4bf1-88c0-a5987bc8b9ed","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"11e60a5a-051d-453f-a190-66387cb35592","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"2c329fbe-bbc4-4030-bf87-d711cad04679","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f4213d4a-bd36-4433-91ae-d7e2b02481d3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"f0cf44ea-6055-48af-805b-aed858102ff9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"d895eaa0-cc0b-4ee4-9b5a-2dfcd89a60b7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"9bc5a7bc-8f52-4db9-96d6-de5e68d52eaa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:26:19.682 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:19.694+0000 7f449a182640 10 librados: watch_flush enter 2024-03-29T11:26:19.682 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:19.694+0000 7f449a182640 10 librados: watch_flush exit 2024-03-29T11:26:19.682 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:19.694+0000 7f449a182640 20 client.4496.objecter shutdown clearing up homeless session... 2024-03-29T11:26:19.682 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:19.694+0000 7f449a182640 10 client.4496.objecter successfully canceled tick 2024-03-29T11:26:19.682 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:19.694+0000 7f449a182640 1 -- 172.21.15.84:0/1703214844 >> v1:172.21.15.179:6816/2981134980 conn(0x7f448006dd50 legacy=0x7f4480070210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:19.682 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:19.694+0000 7f449a182640 10 monclient: shutdown 2024-03-29T11:26:19.682 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:19.694+0000 7f449a182640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:26:19.682 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:19.694+0000 7f449a182640 1 -- 172.21.15.84:0/1703214844 >> v1:172.21.15.84:6790/0 conn(0x7f449413ee40 legacy=0x7f449406de90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:19.682 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:19.694+0000 7f449a182640 1 -- 172.21.15.84:0/1703214844 shutdown_connections 2024-03-29T11:26:19.682 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:19.694+0000 7f449a182640 1 -- 172.21.15.84:0/1703214844 >> 172.21.15.84:0/1703214844 conn(0x7f449406b660 msgr2=0x7f44940a9d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:26:19.682 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:19.695+0000 7f449a182640 1 -- 172.21.15.84:0/1703214844 shutdown_connections 2024-03-29T11:26:19.682 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:19.695+0000 7f449a182640 1 -- 172.21.15.84:0/1703214844 wait complete. 2024-03-29T11:26:19.682 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:19.695+0000 7f449a182640 1 librados: shutdown 2024-03-29T11:26:19.695 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:26:26.501 DEBUG:teuthology.orchestra.run.smithi084:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:26:26.650 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:26.664+0000 7f22254c0640 10 monclient: get_monmap_and_config 2024-03-29T11:26:26.650 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:26.664+0000 7f22254c0640 10 monclient: build_initial_monmap 2024-03-29T11:26:26.650 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:26.664+0000 7f22254c0640 10 monclient: monmap: 2024-03-29T11:26:26.651 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-29T11:26:26.651 INFO:teuthology.misc.health.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:26:26.651 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T11:26:26.665262+0000 2024-03-29T11:26:26.651 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T11:26:26.665262+0000 2024-03-29T11:26:26.651 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-29T11:26:26.651 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:26:26.651 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-29T11:26:26.651 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.noname-c 2024-03-29T11:26:26.651 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-b 2024-03-29T11:26:26.651 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:26:26.652 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:26.666+0000 7f22254c0640 1 Processor -- start 2024-03-29T11:26:26.653 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:26.666+0000 7f22254c0640 1 -- start start 2024-03-29T11:26:26.653 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:26.666+0000 7f22254c0640 10 monclient: init 2024-03-29T11:26:26.653 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:26.667+0000 7f22254c0640 10 monclient: _reopen_session rank -1 2024-03-29T11:26:26.653 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:26.667+0000 7f22254c0640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:26:26.654 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:26.667+0000 7f22254c0640 10 monclient(hunting): picked mon.noname-a con 0x7f22200b0770 addr v1:172.21.15.84:6789/0 2024-03-29T11:26:26.654 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:26.667+0000 7f22254c0640 10 monclient(hunting): picked mon.noname-c con 0x7f222014a700 addr v1:172.21.15.179:6789/0 2024-03-29T11:26:26.654 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:26.667+0000 7f22254c0640 10 monclient(hunting): picked mon.noname-b con 0x7f22200ac5a0 addr v1:172.21.15.84:6790/0 2024-03-29T11:26:26.654 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:26.667+0000 7f22254c0640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f22200aba70 con 0x7f22200b0770 2024-03-29T11:26:26.654 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:26.667+0000 7f22254c0640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f22200aecb0 con 0x7f222014a700 2024-03-29T11:26:26.654 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:26.667+0000 7f22254c0640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f22200af150 con 0x7f22200ac5a0 2024-03-29T11:26:26.654 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:26.667+0000 7f22254c0640 10 monclient(hunting): _renew_subs 2024-03-29T11:26:26.654 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:26.667+0000 7f22254c0640 10 monclient(hunting): authenticate will time out at 1247.961792s 2024-03-29T11:26:26.654 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:26.667+0000 7f221f7fe640 1 --1- >> v1:172.21.15.84:6790/0 conn(0x7f22200ac5a0 0x7f2220151f00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.84:6790/0 says I am v1:172.21.15.84:47878/0 (socket says 172.21.15.84:47878) 2024-03-29T11:26:26.654 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:26.667+0000 7f221f7fe640 1 -- 172.21.15.84:0/101238305 learned_addr learned my addr 172.21.15.84:0/101238305 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-29T11:26:26.655 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:26.669+0000 7f221dffb640 1 -- 172.21.15.84:0/101238305 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2409494268 0 0) 0x7f22200af150 con 0x7f22200ac5a0 2024-03-29T11:26:26.655 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:26.669+0000 7f221dffb640 10 monclient(hunting): _negotiate 2024-03-29T11:26:26.655 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:26.669+0000 7f221dffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:26:26.655 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:26.669+0000 7f221dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:26:26.655 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:26.669+0000 7f221dffb640 10 monclient(hunting): my global_id is 4499 2024-03-29T11:26:26.655 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:26.669+0000 7f221dffb640 1 -- 172.21.15.84:0/101238305 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2208003520 con 0x7f22200ac5a0 2024-03-29T11:26:26.655 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:26.669+0000 7f221dffb640 1 -- 172.21.15.84:0/101238305 <== mon.2 v1:172.21.15.84:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 701769392 0 0) 0x7f2208003520 con 0x7f22200ac5a0 2024-03-29T11:26:26.655 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:26.669+0000 7f221dffb640 1 -- 172.21.15.84:0/101238305 >> v1:172.21.15.179:6789/0 conn(0x7f222014a700 legacy=0x7f22200abe90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:26:26.658 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:26.671+0000 7f221dffb640 1 -- 172.21.15.84:0/101238305 >> v1:172.21.15.84:6789/0 conn(0x7f22200b0770 legacy=0x7f2220149ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:26:26.658 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:26.671+0000 7f221dffb640 10 monclient: _finish_hunting 0 2024-03-29T11:26:26.658 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:26.671+0000 7f221dffb640 1 monclient: found mon.noname-b 2024-03-29T11:26:26.658 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:26.671+0000 7f221dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:26:26.658 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:26.671+0000 7f221dffb640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.84:6790/0 2024-03-29T11:26:26.658 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:26.671+0000 7f221dffb640 1 -- 172.21.15.84:0/101238305 --> v1:172.21.15.84:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22200ac9d0 con 0x7f22200ac5a0 2024-03-29T11:26:26.658 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:26.671+0000 7f221dffb640 10 monclient: _finish_auth 0 2024-03-29T11:26:26.658 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:26.671+0000 7f221dffb640 10 monclient: _check_auth_tickets 2024-03-29T11:26:26.658 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:26.671+0000 7f221dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:26:26.658 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:26.671+0000 7f22254c0640 5 monclient: authenticate success, global_id 4499 2024-03-29T11:26:26.658 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:26.671+0000 7f22254c0640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:26:26.658 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:26.671+0000 7f221dffb640 1 -- 172.21.15.84:0/101238305 <== mon.2 v1:172.21.15.84:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4236868694 0 0) 0x7f2218001a10 con 0x7f22200ac5a0 2024-03-29T11:26:26.658 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:26.671+0000 7f221dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:26.658 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:26.672+0000 7f221dffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:26:26.659 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:26.672+0000 7f221dffb640 10 monclient: dump: 2024-03-29T11:26:26.659 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T11:26:26.659 INFO:teuthology.misc.health.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:26:26.659 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T11:18:53.864025+0000 2024-03-29T11:26:26.659 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T11:18:53.864025+0000 2024-03-29T11:26:26.659 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:26.659 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:26:26.659 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T11:26:26.659 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T11:26:26.659 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T11:26:26.659 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:26:26.659 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:26.672+0000 7f22254c0640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:26:26.661 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:26.673+0000 7f221dffb640 1 -- 172.21.15.84:0/101238305 <== mon.2 v1:172.21.15.84:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2218001ca0 con 0x7f22200ac5a0 2024-03-29T11:26:26.661 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:26.673+0000 7f221dffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:26:26.661 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:26.673+0000 7f22254c0640 10 monclient: get_monmap_and_config success 2024-03-29T11:26:26.661 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:26.673+0000 7f22254c0640 4 set_mon_vals no callback set 2024-03-29T11:26:26.661 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:26.673+0000 7f221dffb640 1 -- 172.21.15.84:0/101238305 <== mon.2 v1:172.21.15.84:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4236868694 0 0) 0x7f2218002030 con 0x7f22200ac5a0 2024-03-29T11:26:26.661 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:26.674+0000 7f221dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:26.661 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:26.674+0000 7f221dffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:26:26.661 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:26.674+0000 7f221dffb640 10 monclient: dump: 2024-03-29T11:26:26.662 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T11:26:26.662 INFO:teuthology.misc.health.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:26:26.662 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T11:18:53.864025+0000 2024-03-29T11:26:26.662 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T11:18:53.864025+0000 2024-03-29T11:26:26.662 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:26.662 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:26:26.662 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T11:26:26.662 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T11:26:26.662 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T11:26:26.662 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:26:26.662 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:26.674+0000 7f22254c0640 10 monclient: shutdown 2024-03-29T11:26:26.662 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:26.674+0000 7f22254c0640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:26:26.662 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:26.674+0000 7f22254c0640 1 -- 172.21.15.84:0/101238305 >> v1:172.21.15.84:6790/0 conn(0x7f22200ac5a0 legacy=0x7f2220151f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:26.662 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:26.675+0000 7f22254c0640 1 -- 172.21.15.84:0/101238305 shutdown_connections 2024-03-29T11:26:26.663 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:26.675+0000 7f22254c0640 1 -- 172.21.15.84:0/101238305 >> 172.21.15.84:0/101238305 conn(0x7f222006b660 msgr2=0x7f222006ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:26:26.663 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:26.675+0000 7f22254c0640 1 -- 172.21.15.84:0/101238305 shutdown_connections 2024-03-29T11:26:26.663 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:26.675+0000 7f22254c0640 1 -- 172.21.15.84:0/101238305 wait complete. 2024-03-29T11:26:26.665 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:26.678+0000 7f22254c0640 10 monclient: build_initial_monmap 2024-03-29T11:26:26.665 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:26.678+0000 7f22254c0640 10 monclient: monmap: 2024-03-29T11:26:26.666 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-29T11:26:26.666 INFO:teuthology.misc.health.smithi084.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:26:26.666 INFO:teuthology.misc.health.smithi084.stderr:last_changed 0.000000 2024-03-29T11:26:26.666 INFO:teuthology.misc.health.smithi084.stderr:created 0.000000 2024-03-29T11:26:26.666 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-29T11:26:26.666 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:26:26.666 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-29T11:26:26.666 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.noname-b 2024-03-29T11:26:26.666 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-c 2024-03-29T11:26:26.666 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:26:26.666 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:26.679+0000 7f22254c0640 1 librados: starting msgr at 2024-03-29T11:26:26.666 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:26.679+0000 7f22254c0640 1 librados: starting objecter 2024-03-29T11:26:26.667 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:26.679+0000 7f22254c0640 1 Processor -- start 2024-03-29T11:26:26.667 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:26.679+0000 7f22254c0640 1 -- start start 2024-03-29T11:26:26.667 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:26.679+0000 7f22254c0640 1 librados: setting wanted keys 2024-03-29T11:26:26.667 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:26.679+0000 7f22254c0640 1 librados: calling monclient init 2024-03-29T11:26:26.667 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:26.679+0000 7f22254c0640 10 monclient: init 2024-03-29T11:26:26.668 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:26.680+0000 7f22254c0640 10 monclient: _reopen_session rank -1 2024-03-29T11:26:26.668 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:26.680+0000 7f22254c0640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:26:26.668 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:26.680+0000 7f22254c0640 10 monclient(hunting): picked mon.noname-b con 0x7f22200ac5a0 addr v1:172.21.15.179:6789/0 2024-03-29T11:26:26.668 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:26.680+0000 7f22254c0640 10 monclient(hunting): picked mon.noname-c con 0x7f22200b0770 addr v1:172.21.15.84:6790/0 2024-03-29T11:26:26.668 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:26.681+0000 7f221e7fc640 1 --1- >> v1:172.21.15.84:6790/0 conn(0x7f22200b0770 0x7f22201268d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.84:6790/0 says I am v1:172.21.15.84:49106/0 (socket says 172.21.15.84:49106) 2024-03-29T11:26:26.668 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:26.681+0000 7f22254c0640 10 monclient(hunting): picked mon.noname-a con 0x7f222014a700 addr v1:172.21.15.84:6789/0 2024-03-29T11:26:26.668 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:26.681+0000 7f22254c0640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f22201283e0 con 0x7f222014a700 2024-03-29T11:26:26.668 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:26.681+0000 7f22254c0640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2220128650 con 0x7f22200ac5a0 2024-03-29T11:26:26.668 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:26.681+0000 7f22254c0640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f22201263b0 con 0x7f22200b0770 2024-03-29T11:26:26.668 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:26.681+0000 7f22254c0640 10 monclient(hunting): _renew_subs 2024-03-29T11:26:26.668 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:26.681+0000 7f22254c0640 10 monclient(hunting): authenticate will time out at 1247.975730s 2024-03-29T11:26:26.669 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:26.681+0000 7f221e7fc640 1 -- 172.21.15.84:0/3246889679 learned_addr learned my addr 172.21.15.84:0/3246889679 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-29T11:26:26.671 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:26.683+0000 7f22077fe640 10 client.?.objecter ms_handle_connect 0x7f22200b0770 2024-03-29T11:26:26.671 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:26.683+0000 7f22077fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:26:26.671 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:26.683+0000 7f22077fe640 1 -- 172.21.15.84:0/3246889679 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 419015497 0 0) 0x7f22201263b0 con 0x7f22200b0770 2024-03-29T11:26:26.671 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:26.683+0000 7f22077fe640 10 monclient(hunting): _negotiate 2024-03-29T11:26:26.671 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:26.683+0000 7f22077fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:26:26.671 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:26.683+0000 7f22077fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:26:26.671 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:26.683+0000 7f22077fe640 10 monclient(hunting): my global_id is 4502 2024-03-29T11:26:26.672 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:26.683+0000 7f22077fe640 1 -- 172.21.15.84:0/3246889679 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f220c004eb0 con 0x7f22200b0770 2024-03-29T11:26:26.672 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:26.683+0000 7f22077fe640 1 -- 172.21.15.84:0/3246889679 <== mon.2 v1:172.21.15.84:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1500280059 0 0) 0x7f220c004eb0 con 0x7f22200b0770 2024-03-29T11:26:26.672 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:26.683+0000 7f22077fe640 1 -- 172.21.15.84:0/3246889679 --> v1:172.21.15.84:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f22201263b0 con 0x7f22200b0770 2024-03-29T11:26:26.672 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:26.684+0000 7f22077fe640 1 -- 172.21.15.84:0/3246889679 <== mon.2 v1:172.21.15.84:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4236868694 0 0) 0x7f22140017d0 con 0x7f22200b0770 2024-03-29T11:26:26.672 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:26.684+0000 7f22077fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:26.672 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:26.684+0000 7f22077fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:26:26.672 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:26.684+0000 7f22077fe640 10 monclient(hunting): dump: 2024-03-29T11:26:26.672 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T11:26:26.672 INFO:teuthology.misc.health.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:26:26.672 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T11:18:53.864025+0000 2024-03-29T11:26:26.672 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T11:18:53.864025+0000 2024-03-29T11:26:26.672 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:26.672 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:26:26.672 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T11:26:26.673 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T11:26:26.673 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T11:26:26.673 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:26:26.673 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:26.684+0000 7f22077fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:26:26.673 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:26.684+0000 7f22077fe640 10 client.?.objecter ms_handle_connect 0x7f222014a700 2024-03-29T11:26:26.673 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:26.684+0000 7f22077fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:26:26.673 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:26.685+0000 7f22077fe640 1 -- 172.21.15.84:0/3246889679 <== mon.0 v1:172.21.15.84:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 972288662 0 0) 0x7f22201283e0 con 0x7f222014a700 2024-03-29T11:26:26.673 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:26.685+0000 7f22077fe640 10 monclient(hunting): _negotiate 2024-03-29T11:26:26.673 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:26.685+0000 7f22077fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:26:26.673 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:26.685+0000 7f22077fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:26:26.673 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:26.685+0000 7f22077fe640 10 monclient(hunting): my global_id is 4482 2024-03-29T11:26:26.673 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:26.685+0000 7f22077fe640 1 -- 172.21.15.84:0/3246889679 --> v1:172.21.15.84:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f220c004eb0 con 0x7f222014a700 2024-03-29T11:26:26.673 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:26.685+0000 7f22077fe640 1 -- 172.21.15.84:0/3246889679 <== mon.0 v1:172.21.15.84:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4027815568 0 0) 0x7f220c004eb0 con 0x7f222014a700 2024-03-29T11:26:26.673 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:26.685+0000 7f22077fe640 1 -- 172.21.15.84:0/3246889679 --> v1:172.21.15.84:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f22201283e0 con 0x7f222014a700 2024-03-29T11:26:26.674 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:26.685+0000 7f22077fe640 1 -- 172.21.15.84:0/3246889679 <== mon.0 v1:172.21.15.84:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4236868694 0 0) 0x7f22180018d0 con 0x7f222014a700 2024-03-29T11:26:26.674 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:26.685+0000 7f22077fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:26.674 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:26.685+0000 7f22077fe640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:26:26.674 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:26.685+0000 7f22077fe640 10 monclient(hunting): dump: 2024-03-29T11:26:26.674 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T11:26:26.674 INFO:teuthology.misc.health.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:26:26.674 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T11:18:53.864025+0000 2024-03-29T11:26:26.674 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T11:18:53.864025+0000 2024-03-29T11:26:26.674 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:26.674 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:26:26.674 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T11:26:26.674 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T11:26:26.674 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T11:26:26.674 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:26:26.675 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:26.685+0000 7f22077fe640 1 -- 172.21.15.84:0/3246889679 <== mon.2 v1:172.21.15.84:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 447800211 0 0) 0x7f22201263b0 con 0x7f22200b0770 2024-03-29T11:26:26.675 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:26.685+0000 7f22077fe640 1 -- 172.21.15.84:0/3246889679 >> v1:172.21.15.179:6789/0 conn(0x7f22200ac5a0 legacy=0x7f2220144f60 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:26:26.676 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:26.687+0000 7f22077fe640 1 -- 172.21.15.84:0/3246889679 >> v1:172.21.15.84:6789/0 conn(0x7f222014a700 legacy=0x7f2220126fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:26.676 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:26.687+0000 7f22077fe640 10 monclient: _finish_hunting 0 2024-03-29T11:26:26.676 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:26.687+0000 7f22077fe640 1 monclient: found mon.c 2024-03-29T11:26:26.676 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:26.687+0000 7f22077fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:26:26.676 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:26.687+0000 7f22077fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-29T11:26:26.676 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:26.687+0000 7f22077fe640 1 -- 172.21.15.84:0/3246889679 --> v1:172.21.15.84:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2220125ff0 con 0x7f22200b0770 2024-03-29T11:26:26.676 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:26.687+0000 7f22077fe640 10 monclient: _finish_auth 0 2024-03-29T11:26:26.676 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:26.687+0000 7f22077fe640 10 monclient: _check_auth_tickets 2024-03-29T11:26:26.676 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:26.687+0000 7f22077fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:26:26.676 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:26.687+0000 7f22254c0640 5 monclient: authenticate success, global_id 4502 2024-03-29T11:26:26.676 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:26.687+0000 7f22254c0640 10 monclient: _renew_subs 2024-03-29T11:26:26.677 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:26.687+0000 7f22254c0640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-29T11:26:26.677 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:26.687+0000 7f22254c0640 1 -- 172.21.15.84:0/3246889679 --> v1:172.21.15.84:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f22201dfa00 con 0x7f22200b0770 2024-03-29T11:26:26.677 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:26.687+0000 7f22254c0640 10 client.4502.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:26:26.677 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:26.687+0000 7f22254c0640 10 monclient: _renew_subs 2024-03-29T11:26:26.677 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:26.687+0000 7f22254c0640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-29T11:26:26.677 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:26.687+0000 7f22254c0640 1 -- 172.21.15.84:0/3246889679 --> v1:172.21.15.84:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f22201df1d0 con 0x7f22200b0770 2024-03-29T11:26:26.677 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:26.687+0000 7f22254c0640 1 librados: init done 2024-03-29T11:26:26.677 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:26.689+0000 7f22254c0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:26:26.677 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:26.689+0000 7f22254c0640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:26:26.677 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:26.689+0000 7f22254c0640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-29T11:26:26.677 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:26.689+0000 7f22254c0640 1 -- 172.21.15.84:0/3246889679 --> v1:172.21.15.84:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2220126540 con 0x7f22200b0770 2024-03-29T11:26:26.679 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:26.690+0000 7f22077fe640 1 -- 172.21.15.84:0/3246889679 <== mon.2 v1:172.21.15.84:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2214001930 con 0x7f22200b0770 2024-03-29T11:26:26.679 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:26.690+0000 7f22077fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:26:26.679 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:26.690+0000 7f22077fe640 1 -- 172.21.15.84:0/3246889679 <== mon.2 v1:172.21.15.84:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4236868694 0 0) 0x7f2214004a40 con 0x7f22200b0770 2024-03-29T11:26:26.680 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:26.690+0000 7f22077fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:26.680 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:26.690+0000 7f22077fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:26:26.680 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:26.690+0000 7f22077fe640 10 monclient: dump: 2024-03-29T11:26:26.680 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T11:26:26.680 INFO:teuthology.misc.health.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:26:26.680 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T11:18:53.864025+0000 2024-03-29T11:26:26.680 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T11:18:53.864025+0000 2024-03-29T11:26:26.680 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:26.680 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:26:26.680 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T11:26:26.680 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T11:26:26.680 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T11:26:26.680 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:26:26.680 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:26.690+0000 7f22077fe640 1 -- 172.21.15.84:0/3246889679 <== mon.2 v1:172.21.15.84:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4033942909 0 0) 0x7f221401aad0 con 0x7f22200b0770 2024-03-29T11:26:26.681 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:26.691+0000 7f22077fe640 1 -- 172.21.15.84:0/3246889679 <== mon.2 v1:172.21.15.84:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2808139587 0 0) 0x7f2214084b70 con 0x7f22200b0770 2024-03-29T11:26:26.681 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:26.691+0000 7f22077fe640 10 client.4502.objecter ms_dispatch 0x7f2220000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:26:26.681 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:26.691+0000 7f22077fe640 3 client.4502.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:26:26.681 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:26.691+0000 7f22077fe640 3 client.4502.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:26:26.681 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:26.691+0000 7f22077fe640 20 client.4502.objecter dump_active .. 0 homeless 2024-03-29T11:26:26.681 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:26.691+0000 7f221d7fa640 4 set_mon_vals no callback set 2024-03-29T11:26:26.681 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:26.693+0000 7f221effd640 10 monclient: get_auth_request con 0x7f220c06de60 auth_method 0 2024-03-29T11:26:26.681 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:26.695+0000 7f22077fe640 10 client.4502.objecter ms_handle_connect 0x7f220c06de60 2024-03-29T11:26:26.688 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:26.702+0000 7f22077fe640 1 -- 172.21.15.84:0/3246889679 <== mon.2 v1:172.21.15.84:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f2214050300 con 0x7f22200b0770 2024-03-29T11:26:26.688 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:26.702+0000 7f22077fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:26:26.688 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:26.702+0000 7f22077fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:26:27.016 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:27.030+0000 7f22254c0640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:26:27.016 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:27.030+0000 7f22254c0640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:26:27.016 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:27.030+0000 7f22254c0640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-29T11:26:27.017 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:27.030+0000 7f22254c0640 1 -- 172.21.15.84:0/3246889679 --> v1:172.21.15.84:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2220075f10 con 0x7f22200b0770 2024-03-29T11:26:27.017 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:27.031+0000 7f22077fe640 1 -- 172.21.15.84:0/3246889679 <== mon.2 v1:172.21.15.84:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4209477549) 0x7f2214052400 con 0x7f22200b0770 2024-03-29T11:26:27.017 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:27.031+0000 7f22077fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:26:27.018 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:27.031+0000 7f22077fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:26:27.022 INFO:teuthology.misc.health.smithi084.stdout: 2024-03-29T11:26:27.022 INFO:teuthology.misc.health.smithi084.stdout:{"epoch":10,"fsid":"2c3e002f-9eb0-4860-853b-4ef1ef562213","created":"2024-03-29T11:19:32.029169+0000","modified":"2024-03-29T11:19:34.200299+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:34.200299+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"5c39e276-2034-4716-bbc0-57a634eefa92","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"cd580e99-109f-4bf1-88c0-a5987bc8b9ed","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"11e60a5a-051d-453f-a190-66387cb35592","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"2c329fbe-bbc4-4030-bf87-d711cad04679","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f4213d4a-bd36-4433-91ae-d7e2b02481d3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"f0cf44ea-6055-48af-805b-aed858102ff9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"d895eaa0-cc0b-4ee4-9b5a-2dfcd89a60b7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"9bc5a7bc-8f52-4db9-96d6-de5e68d52eaa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:26:27.023 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:27.036+0000 7f22254c0640 10 librados: watch_flush enter 2024-03-29T11:26:27.023 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:27.036+0000 7f22254c0640 10 librados: watch_flush exit 2024-03-29T11:26:27.023 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:27.036+0000 7f22254c0640 20 client.4502.objecter shutdown clearing up homeless session... 2024-03-29T11:26:27.023 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:27.036+0000 7f22254c0640 10 client.4502.objecter successfully canceled tick 2024-03-29T11:26:27.023 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:27.037+0000 7f22254c0640 1 -- 172.21.15.84:0/3246889679 >> v1:172.21.15.179:6816/2981134980 conn(0x7f220c06de60 legacy=0x7f220c070320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:27.023 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:27.037+0000 7f22254c0640 10 monclient: shutdown 2024-03-29T11:26:27.024 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:27.037+0000 7f22254c0640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:26:27.024 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:27.037+0000 7f22254c0640 1 -- 172.21.15.84:0/3246889679 >> v1:172.21.15.84:6790/0 conn(0x7f22200b0770 legacy=0x7f22201268d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:27.024 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:27.037+0000 7f22254c0640 1 -- 172.21.15.84:0/3246889679 shutdown_connections 2024-03-29T11:26:27.024 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:27.037+0000 7f22254c0640 1 -- 172.21.15.84:0/3246889679 >> 172.21.15.84:0/3246889679 conn(0x7f222006b660 msgr2=0x7f22200b2d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:26:27.024 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:27.037+0000 7f22254c0640 1 -- 172.21.15.84:0/3246889679 shutdown_connections 2024-03-29T11:26:27.024 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:27.037+0000 7f22254c0640 1 -- 172.21.15.84:0/3246889679 wait complete. 2024-03-29T11:26:27.024 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:27.037+0000 7f22254c0640 1 librados: shutdown 2024-03-29T11:26:27.043 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:26:33.851 DEBUG:teuthology.orchestra.run.smithi084:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:26:33.999 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:34.012+0000 7fc322a3e640 10 monclient: get_monmap_and_config 2024-03-29T11:26:33.999 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:34.012+0000 7fc322a3e640 10 monclient: build_initial_monmap 2024-03-29T11:26:33.999 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:34.012+0000 7fc322a3e640 10 monclient: monmap: 2024-03-29T11:26:33.999 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-29T11:26:33.999 INFO:teuthology.misc.health.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:26:33.999 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T11:26:34.013876+0000 2024-03-29T11:26:33.999 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T11:26:34.013876+0000 2024-03-29T11:26:33.999 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-29T11:26:34.000 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:26:34.000 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-29T11:26:34.000 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.noname-c 2024-03-29T11:26:34.000 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-b 2024-03-29T11:26:34.000 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:26:34.001 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:34.015+0000 7fc322a3e640 1 Processor -- start 2024-03-29T11:26:34.002 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:34.015+0000 7fc322a3e640 1 -- start start 2024-03-29T11:26:34.002 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:34.015+0000 7fc322a3e640 10 monclient: init 2024-03-29T11:26:34.003 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:34.016+0000 7fc322a3e640 10 monclient: _reopen_session rank -1 2024-03-29T11:26:34.003 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:34.016+0000 7fc322a3e640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:26:34.003 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:34.016+0000 7fc322a3e640 10 monclient(hunting): picked mon.noname-b con 0x7fc31c0b0860 addr v1:172.21.15.84:6790/0 2024-03-29T11:26:34.003 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:34.017+0000 7fc31bfff640 1 --1- >> v1:172.21.15.84:6790/0 conn(0x7fc31c0b0860 0x7fc31c13e820 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.84:6790/0 says I am v1:172.21.15.84:52222/0 (socket says 172.21.15.84:52222) 2024-03-29T11:26:34.003 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:34.017+0000 7fc31bfff640 1 -- 172.21.15.84:0/1269648405 learned_addr learned my addr 172.21.15.84:0/1269648405 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-29T11:26:34.005 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:34.017+0000 7fc322a3e640 10 monclient(hunting): picked mon.noname-a con 0x7fc31c13ef30 addr v1:172.21.15.84:6789/0 2024-03-29T11:26:34.006 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:34.017+0000 7fc322a3e640 10 monclient(hunting): picked mon.noname-c con 0x7fc31c0af8e0 addr v1:172.21.15.179:6789/0 2024-03-29T11:26:34.007 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:34.020+0000 7fc322a3e640 1 -- 172.21.15.84:0/1269648405 --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc31c0b0c40 con 0x7fc31c13ef30 2024-03-29T11:26:34.007 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:34.020+0000 7fc322a3e640 1 -- 172.21.15.84:0/1269648405 --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc31c0aed10 con 0x7fc31c0af8e0 2024-03-29T11:26:34.007 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:34.020+0000 7fc322a3e640 1 -- 172.21.15.84:0/1269648405 --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc31c0afd50 con 0x7fc31c0b0860 2024-03-29T11:26:34.007 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:34.020+0000 7fc322a3e640 10 monclient(hunting): _renew_subs 2024-03-29T11:26:34.007 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:34.020+0000 7fc322a3e640 10 monclient(hunting): authenticate will time out at 1255.315019s 2024-03-29T11:26:34.008 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:34.021+0000 7fc31affd640 1 -- 172.21.15.84:0/1269648405 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3305024151 0 0) 0x7fc31c0afd50 con 0x7fc31c0b0860 2024-03-29T11:26:34.009 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:34.021+0000 7fc31affd640 10 monclient(hunting): _negotiate 2024-03-29T11:26:34.009 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:34.021+0000 7fc31affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:26:34.009 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:34.021+0000 7fc31affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:26:34.009 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:34.021+0000 7fc31affd640 10 monclient(hunting): my global_id is 4505 2024-03-29T11:26:34.010 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:34.021+0000 7fc31affd640 1 -- 172.21.15.84:0/1269648405 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc304003520 con 0x7fc31c0b0860 2024-03-29T11:26:34.010 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:34.022+0000 7fc31affd640 1 -- 172.21.15.84:0/1269648405 <== mon.0 v1:172.21.15.84:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1911731563 0 0) 0x7fc31c0b0c40 con 0x7fc31c13ef30 2024-03-29T11:26:34.010 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:34.022+0000 7fc31affd640 10 monclient(hunting): _negotiate 2024-03-29T11:26:34.010 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:34.022+0000 7fc31affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:26:34.010 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:34.022+0000 7fc31affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:26:34.010 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:34.022+0000 7fc31affd640 10 monclient(hunting): my global_id is 4485 2024-03-29T11:26:34.010 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:34.022+0000 7fc31affd640 1 -- 172.21.15.84:0/1269648405 --> v1:172.21.15.84:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc31c0afd50 con 0x7fc31c13ef30 2024-03-29T11:26:34.010 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:34.022+0000 7fc31affd640 1 -- 172.21.15.84:0/1269648405 <== mon.2 v1:172.21.15.84:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1456500396 0 0) 0x7fc304003520 con 0x7fc31c0b0860 2024-03-29T11:26:34.011 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:34.022+0000 7fc31affd640 1 -- 172.21.15.84:0/1269648405 >> v1:172.21.15.179:6789/0 conn(0x7fc31c0af8e0 legacy=0x7fc31c149a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:34.011 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:34.023+0000 7fc31affd640 1 -- 172.21.15.84:0/1269648405 >> v1:172.21.15.84:6789/0 conn(0x7fc31c13ef30 legacy=0x7fc31c0af1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:34.012 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:34.023+0000 7fc31affd640 10 monclient: _finish_hunting 0 2024-03-29T11:26:34.012 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:34.023+0000 7fc31affd640 1 monclient: found mon.noname-b 2024-03-29T11:26:34.012 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:34.023+0000 7fc31affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:26:34.012 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:34.023+0000 7fc31affd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.84:6790/0 2024-03-29T11:26:34.012 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:34.023+0000 7fc31affd640 1 -- 172.21.15.84:0/1269648405 --> v1:172.21.15.84:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc31c0ab1b0 con 0x7fc31c0b0860 2024-03-29T11:26:34.012 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:34.023+0000 7fc31affd640 10 monclient: _finish_auth 0 2024-03-29T11:26:34.012 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:34.023+0000 7fc31affd640 10 monclient: _check_auth_tickets 2024-03-29T11:26:34.012 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:34.023+0000 7fc31affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:26:34.012 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:34.023+0000 7fc31affd640 1 -- 172.21.15.84:0/1269648405 <== mon.2 v1:172.21.15.84:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4236868694 0 0) 0x7fc30c001690 con 0x7fc31c0b0860 2024-03-29T11:26:34.012 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:34.023+0000 7fc31affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:34.012 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:34.023+0000 7fc31affd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:26:34.012 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:34.023+0000 7fc31affd640 10 monclient: dump: 2024-03-29T11:26:34.012 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T11:26:34.012 INFO:teuthology.misc.health.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:26:34.013 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T11:18:53.864025+0000 2024-03-29T11:26:34.013 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T11:18:53.864025+0000 2024-03-29T11:26:34.013 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:34.013 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:26:34.013 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T11:26:34.013 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T11:26:34.013 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T11:26:34.013 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:26:34.013 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:34.023+0000 7fc31affd640 1 -- 172.21.15.84:0/1269648405 <== mon.2 v1:172.21.15.84:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc30c0018b0 con 0x7fc31c0b0860 2024-03-29T11:26:34.013 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:34.023+0000 7fc31affd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:26:34.013 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:34.023+0000 7fc31affd640 1 -- 172.21.15.84:0/1269648405 <== mon.2 v1:172.21.15.84:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4236868694 0 0) 0x7fc30c001c40 con 0x7fc31c0b0860 2024-03-29T11:26:34.013 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:34.023+0000 7fc31affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:34.013 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:34.023+0000 7fc31affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:26:34.013 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:34.023+0000 7fc31affd640 10 monclient: dump: 2024-03-29T11:26:34.014 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T11:26:34.014 INFO:teuthology.misc.health.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:26:34.014 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T11:18:53.864025+0000 2024-03-29T11:26:34.014 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T11:18:53.864025+0000 2024-03-29T11:26:34.014 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:34.014 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:26:34.014 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T11:26:34.014 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T11:26:34.014 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T11:26:34.014 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:26:34.014 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:34.023+0000 7fc322a3e640 5 monclient: authenticate success, global_id 4505 2024-03-29T11:26:34.014 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:34.023+0000 7fc322a3e640 10 monclient: get_monmap_and_config success 2024-03-29T11:26:34.014 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:34.023+0000 7fc322a3e640 4 set_mon_vals no callback set 2024-03-29T11:26:34.014 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:34.024+0000 7fc322a3e640 10 monclient: shutdown 2024-03-29T11:26:34.015 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:34.025+0000 7fc322a3e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:26:34.015 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:34.025+0000 7fc322a3e640 1 -- 172.21.15.84:0/1269648405 >> v1:172.21.15.84:6790/0 conn(0x7fc31c0b0860 legacy=0x7fc31c13e820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:34.015 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:34.029+0000 7fc322a3e640 1 -- 172.21.15.84:0/1269648405 shutdown_connections 2024-03-29T11:26:34.015 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:34.029+0000 7fc322a3e640 1 -- 172.21.15.84:0/1269648405 >> 172.21.15.84:0/1269648405 conn(0x7fc31c06b660 msgr2=0x7fc31c06ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:26:34.015 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:34.029+0000 7fc322a3e640 1 -- 172.21.15.84:0/1269648405 shutdown_connections 2024-03-29T11:26:34.015 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:34.029+0000 7fc322a3e640 1 -- 172.21.15.84:0/1269648405 wait complete. 2024-03-29T11:26:34.016 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:34.030+0000 7fc322a3e640 10 monclient: build_initial_monmap 2024-03-29T11:26:34.016 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:34.030+0000 7fc322a3e640 10 monclient: monmap: 2024-03-29T11:26:34.016 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-29T11:26:34.016 INFO:teuthology.misc.health.smithi084.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:26:34.016 INFO:teuthology.misc.health.smithi084.stderr:last_changed 0.000000 2024-03-29T11:26:34.017 INFO:teuthology.misc.health.smithi084.stderr:created 0.000000 2024-03-29T11:26:34.017 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-29T11:26:34.017 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:26:34.017 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-29T11:26:34.017 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.noname-b 2024-03-29T11:26:34.017 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-c 2024-03-29T11:26:34.017 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:26:34.019 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:34.031+0000 7fc322a3e640 1 librados: starting msgr at 2024-03-29T11:26:34.019 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:34.031+0000 7fc322a3e640 1 librados: starting objecter 2024-03-29T11:26:34.019 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:34.031+0000 7fc322a3e640 1 Processor -- start 2024-03-29T11:26:34.019 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:34.031+0000 7fc322a3e640 1 -- start start 2024-03-29T11:26:34.019 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:34.031+0000 7fc322a3e640 1 librados: setting wanted keys 2024-03-29T11:26:34.019 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:34.031+0000 7fc322a3e640 1 librados: calling monclient init 2024-03-29T11:26:34.019 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:34.031+0000 7fc322a3e640 10 monclient: init 2024-03-29T11:26:34.024 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:34.032+0000 7fc322a3e640 10 monclient: _reopen_session rank -1 2024-03-29T11:26:34.024 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:34.032+0000 7fc322a3e640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:26:34.024 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:34.032+0000 7fc322a3e640 10 monclient(hunting): picked mon.noname-b con 0x7fc31c0af8e0 addr v1:172.21.15.179:6789/0 2024-03-29T11:26:34.024 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:34.032+0000 7fc322a3e640 10 monclient(hunting): picked mon.noname-a con 0x7fc31c0b0860 addr v1:172.21.15.84:6789/0 2024-03-29T11:26:34.024 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:34.032+0000 7fc322a3e640 10 monclient(hunting): picked mon.noname-c con 0x7fc31c13ef30 addr v1:172.21.15.84:6790/0 2024-03-29T11:26:34.024 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:34.032+0000 7fc322a3e640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc31c0b3fe0 con 0x7fc31c0b0860 2024-03-29T11:26:34.024 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:34.032+0000 7fc322a3e640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc31c0b42c0 con 0x7fc31c0af8e0 2024-03-29T11:26:34.025 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:34.032+0000 7fc322a3e640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc31c1dbd80 con 0x7fc31c13ef30 2024-03-29T11:26:34.025 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:34.032+0000 7fc322a3e640 10 monclient(hunting): _renew_subs 2024-03-29T11:26:34.025 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:34.032+0000 7fc322a3e640 10 monclient(hunting): authenticate will time out at 1255.326769s 2024-03-29T11:26:34.025 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:34.032+0000 7fc320fdb640 1 --1- >> v1:172.21.15.84:6790/0 conn(0x7fc31c13ef30 0x7fc31c1e7420 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.84:6790/0 says I am v1:172.21.15.84:53392/0 (socket says 172.21.15.84:53392) 2024-03-29T11:26:34.025 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:34.032+0000 7fc320fdb640 1 -- 172.21.15.84:0/2816566212 learned_addr learned my addr 172.21.15.84:0/2816566212 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-29T11:26:34.025 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:34.032+0000 7fc318ff9640 10 client.?.objecter ms_handle_connect 0x7fc31c13ef30 2024-03-29T11:26:34.025 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:34.032+0000 7fc318ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:26:34.025 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:34.032+0000 7fc318ff9640 1 -- 172.21.15.84:0/2816566212 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2508292106 0 0) 0x7fc31c1dbd80 con 0x7fc31c13ef30 2024-03-29T11:26:34.025 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:34.033+0000 7fc318ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:26:34.025 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:34.033+0000 7fc318ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:26:34.025 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:34.033+0000 7fc318ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:26:34.025 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:34.033+0000 7fc318ff9640 10 monclient(hunting): my global_id is 4508 2024-03-29T11:26:34.025 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:34.033+0000 7fc318ff9640 1 -- 172.21.15.84:0/2816566212 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc308004eb0 con 0x7fc31c13ef30 2024-03-29T11:26:34.026 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:34.033+0000 7fc318ff9640 1 -- 172.21.15.84:0/2816566212 <== mon.2 v1:172.21.15.84:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3047750353 0 0) 0x7fc308004eb0 con 0x7fc31c13ef30 2024-03-29T11:26:34.026 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:34.033+0000 7fc318ff9640 1 -- 172.21.15.84:0/2816566212 --> v1:172.21.15.84:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc31c1dbd80 con 0x7fc31c13ef30 2024-03-29T11:26:34.026 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:34.033+0000 7fc318ff9640 1 -- 172.21.15.84:0/2816566212 <== mon.2 v1:172.21.15.84:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4236868694 0 0) 0x7fc314001cd0 con 0x7fc31c13ef30 2024-03-29T11:26:34.026 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:34.033+0000 7fc318ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:34.026 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:34.033+0000 7fc318ff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:26:34.026 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:34.033+0000 7fc318ff9640 10 monclient(hunting): dump: 2024-03-29T11:26:34.026 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T11:26:34.026 INFO:teuthology.misc.health.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:26:34.026 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T11:18:53.864025+0000 2024-03-29T11:26:34.026 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T11:18:53.864025+0000 2024-03-29T11:26:34.026 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:34.026 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:26:34.026 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T11:26:34.026 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T11:26:34.027 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T11:26:34.027 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:26:34.027 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:34.033+0000 7fc318ff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:26:34.027 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:34.033+0000 7fc318ff9640 1 -- 172.21.15.84:0/2816566212 <== mon.2 v1:172.21.15.84:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2765340455 0 0) 0x7fc31c1dbd80 con 0x7fc31c13ef30 2024-03-29T11:26:34.027 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:34.033+0000 7fc318ff9640 1 -- 172.21.15.84:0/2816566212 >> v1:172.21.15.179:6789/0 conn(0x7fc31c0af8e0 legacy=0x7fc31c1dc2a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:26:34.027 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:34.036+0000 7fc318ff9640 1 -- 172.21.15.84:0/2816566212 >> v1:172.21.15.84:6789/0 conn(0x7fc31c0b0860 legacy=0x7fc31c1daeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:34.027 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:34.036+0000 7fc318ff9640 10 monclient: _finish_hunting 0 2024-03-29T11:26:34.027 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:34.036+0000 7fc318ff9640 1 monclient: found mon.c 2024-03-29T11:26:34.027 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:34.036+0000 7fc318ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:26:34.027 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:34.036+0000 7fc318ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-29T11:26:34.027 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:34.036+0000 7fc318ff9640 1 -- 172.21.15.84:0/2816566212 --> v1:172.21.15.84:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc31c0b1bf0 con 0x7fc31c13ef30 2024-03-29T11:26:34.028 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:34.036+0000 7fc318ff9640 10 monclient: _finish_auth 0 2024-03-29T11:26:34.028 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:34.036+0000 7fc318ff9640 10 monclient: _check_auth_tickets 2024-03-29T11:26:34.028 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:34.036+0000 7fc318ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:26:34.028 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:34.036+0000 7fc318ff9640 1 -- 172.21.15.84:0/2816566212 <== mon.2 v1:172.21.15.84:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc314001fd0 con 0x7fc31c13ef30 2024-03-29T11:26:34.028 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:34.036+0000 7fc318ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:26:34.028 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:34.036+0000 7fc318ff9640 1 -- 172.21.15.84:0/2816566212 <== mon.2 v1:172.21.15.84:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4236868694 0 0) 0x7fc314003de0 con 0x7fc31c13ef30 2024-03-29T11:26:34.028 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:34.036+0000 7fc318ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:34.028 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:34.036+0000 7fc318ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:26:34.028 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:34.036+0000 7fc318ff9640 10 monclient: dump: 2024-03-29T11:26:34.029 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T11:26:34.029 INFO:teuthology.misc.health.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:26:34.029 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T11:18:53.864025+0000 2024-03-29T11:26:34.029 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T11:18:53.864025+0000 2024-03-29T11:26:34.029 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:34.029 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:26:34.029 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T11:26:34.029 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T11:26:34.029 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T11:26:34.029 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:26:34.029 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:34.036+0000 7fc319ffb640 4 set_mon_vals no callback set 2024-03-29T11:26:34.031 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:34.038+0000 7fc322a3e640 5 monclient: authenticate success, global_id 4508 2024-03-29T11:26:34.031 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:34.038+0000 7fc322a3e640 10 monclient: _renew_subs 2024-03-29T11:26:34.031 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:34.038+0000 7fc322a3e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-29T11:26:34.031 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:34.038+0000 7fc322a3e640 1 -- 172.21.15.84:0/2816566212 --> v1:172.21.15.84:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc31c1dc9b0 con 0x7fc31c13ef30 2024-03-29T11:26:34.031 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:34.038+0000 7fc322a3e640 10 client.4508.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:26:34.031 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:34.038+0000 7fc322a3e640 10 monclient: _renew_subs 2024-03-29T11:26:34.031 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:34.038+0000 7fc322a3e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-29T11:26:34.031 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:34.038+0000 7fc322a3e640 1 -- 172.21.15.84:0/2816566212 --> v1:172.21.15.84:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc31c13a0f0 con 0x7fc31c13ef30 2024-03-29T11:26:34.031 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:34.038+0000 7fc322a3e640 1 librados: init done 2024-03-29T11:26:34.031 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:34.039+0000 7fc322a3e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:26:34.031 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:34.040+0000 7fc318ff9640 1 -- 172.21.15.84:0/2816566212 <== mon.2 v1:172.21.15.84:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4033942909 0 0) 0x7fc314003f40 con 0x7fc31c13ef30 2024-03-29T11:26:34.031 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:34.040+0000 7fc322a3e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:26:34.032 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:34.040+0000 7fc322a3e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-29T11:26:34.032 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:34.040+0000 7fc322a3e640 1 -- 172.21.15.84:0/2816566212 --> v1:172.21.15.84:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc31c1dbbe0 con 0x7fc31c13ef30 2024-03-29T11:26:34.032 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:34.040+0000 7fc318ff9640 1 -- 172.21.15.84:0/2816566212 <== mon.2 v1:172.21.15.84:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2808139587 0 0) 0x7fc314084e50 con 0x7fc31c13ef30 2024-03-29T11:26:34.032 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:34.040+0000 7fc318ff9640 10 client.4508.objecter ms_dispatch 0x7fc31c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:26:34.032 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:34.040+0000 7fc318ff9640 3 client.4508.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:26:34.032 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:34.040+0000 7fc318ff9640 3 client.4508.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:26:34.032 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:34.040+0000 7fc318ff9640 20 client.4508.objecter dump_active .. 0 homeless 2024-03-29T11:26:34.032 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:34.040+0000 7fc31bfff640 10 monclient: get_auth_request con 0x7fc30806dd20 auth_method 0 2024-03-29T11:26:34.032 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:34.041+0000 7fc318ff9640 10 client.4508.objecter ms_handle_connect 0x7fc30806dd20 2024-03-29T11:26:34.036 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:34.049+0000 7fc318ff9640 1 -- 172.21.15.84:0/2816566212 <== mon.2 v1:172.21.15.84:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fc31c1dbbe0 con 0x7fc31c13ef30 2024-03-29T11:26:34.036 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:34.049+0000 7fc318ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:26:34.036 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:34.049+0000 7fc318ff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:26:34.373 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:34.387+0000 7fc322a3e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:26:34.373 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:34.387+0000 7fc322a3e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:26:34.373 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:34.387+0000 7fc322a3e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-29T11:26:34.373 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:34.387+0000 7fc322a3e640 1 -- 172.21.15.84:0/2816566212 --> v1:172.21.15.84:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc31c1dbbe0 con 0x7fc31c13ef30 2024-03-29T11:26:34.374 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:34.388+0000 7fc318ff9640 1 -- 172.21.15.84:0/2816566212 <== mon.2 v1:172.21.15.84:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4209477549) 0x7fc31c1dbbe0 con 0x7fc31c13ef30 2024-03-29T11:26:34.374 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:34.388+0000 7fc318ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:26:34.374 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:34.388+0000 7fc318ff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:26:34.374 INFO:teuthology.misc.health.smithi084.stdout: 2024-03-29T11:26:34.374 INFO:teuthology.misc.health.smithi084.stdout:{"epoch":10,"fsid":"2c3e002f-9eb0-4860-853b-4ef1ef562213","created":"2024-03-29T11:19:32.029169+0000","modified":"2024-03-29T11:19:34.200299+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:34.200299+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"5c39e276-2034-4716-bbc0-57a634eefa92","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"cd580e99-109f-4bf1-88c0-a5987bc8b9ed","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"11e60a5a-051d-453f-a190-66387cb35592","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"2c329fbe-bbc4-4030-bf87-d711cad04679","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f4213d4a-bd36-4433-91ae-d7e2b02481d3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"f0cf44ea-6055-48af-805b-aed858102ff9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"d895eaa0-cc0b-4ee4-9b5a-2dfcd89a60b7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"9bc5a7bc-8f52-4db9-96d6-de5e68d52eaa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:26:34.377 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:34.391+0000 7fc322a3e640 10 librados: watch_flush enter 2024-03-29T11:26:34.378 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:34.391+0000 7fc322a3e640 10 librados: watch_flush exit 2024-03-29T11:26:34.378 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:34.391+0000 7fc322a3e640 20 client.4508.objecter shutdown clearing up homeless session... 2024-03-29T11:26:34.378 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:34.391+0000 7fc322a3e640 10 client.4508.objecter successfully canceled tick 2024-03-29T11:26:34.379 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:34.392+0000 7fc322a3e640 1 -- 172.21.15.84:0/2816566212 >> v1:172.21.15.179:6816/2981134980 conn(0x7fc30806dd20 legacy=0x7fc3080701e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:34.379 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:34.392+0000 7fc322a3e640 10 monclient: shutdown 2024-03-29T11:26:34.379 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:34.392+0000 7fc322a3e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:26:34.379 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:34.392+0000 7fc322a3e640 1 -- 172.21.15.84:0/2816566212 >> v1:172.21.15.84:6790/0 conn(0x7fc31c13ef30 legacy=0x7fc31c1e7420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:34.379 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:34.392+0000 7fc322a3e640 1 -- 172.21.15.84:0/2816566212 shutdown_connections 2024-03-29T11:26:34.379 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:34.392+0000 7fc322a3e640 1 -- 172.21.15.84:0/2816566212 >> 172.21.15.84:0/2816566212 conn(0x7fc31c06b660 msgr2=0x7fc31c06ba40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:26:34.379 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:34.392+0000 7fc322a3e640 1 -- 172.21.15.84:0/2816566212 shutdown_connections 2024-03-29T11:26:34.379 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:34.392+0000 7fc322a3e640 1 -- 172.21.15.84:0/2816566212 wait complete. 2024-03-29T11:26:34.379 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:34.392+0000 7fc322a3e640 1 librados: shutdown 2024-03-29T11:26:34.396 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:26:41.203 DEBUG:teuthology.orchestra.run.smithi084:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:26:41.350 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:41.364+0000 7ff15b59e640 10 monclient: get_monmap_and_config 2024-03-29T11:26:41.351 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:41.364+0000 7ff15b59e640 10 monclient: build_initial_monmap 2024-03-29T11:26:41.351 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:41.364+0000 7ff15b59e640 10 monclient: monmap: 2024-03-29T11:26:41.351 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-29T11:26:41.351 INFO:teuthology.misc.health.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:26:41.351 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T11:26:41.364968+0000 2024-03-29T11:26:41.351 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T11:26:41.364968+0000 2024-03-29T11:26:41.351 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-29T11:26:41.351 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:26:41.351 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-29T11:26:41.351 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.noname-c 2024-03-29T11:26:41.351 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-b 2024-03-29T11:26:41.351 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:26:41.352 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:41.364+0000 7ff15b59e640 1 Processor -- start 2024-03-29T11:26:41.352 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:41.365+0000 7ff15b59e640 1 -- start start 2024-03-29T11:26:41.352 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:41.365+0000 7ff15b59e640 10 monclient: init 2024-03-29T11:26:41.353 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:41.365+0000 7ff15b59e640 10 monclient: _reopen_session rank -1 2024-03-29T11:26:41.353 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:41.365+0000 7ff15b59e640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:26:41.353 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:41.367+0000 7ff15b59e640 10 monclient(hunting): picked mon.noname-a con 0x7ff15c0b2800 addr v1:172.21.15.84:6789/0 2024-03-29T11:26:41.353 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:41.367+0000 7ff15b59e640 10 monclient(hunting): picked mon.noname-c con 0x7ff15c0a9c00 addr v1:172.21.15.179:6789/0 2024-03-29T11:26:41.354 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:41.367+0000 7ff15b59e640 10 monclient(hunting): picked mon.noname-b con 0x7ff15c0b38c0 addr v1:172.21.15.84:6790/0 2024-03-29T11:26:41.354 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:41.367+0000 7ff15b59e640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff15c141eb0 con 0x7ff15c0b2800 2024-03-29T11:26:41.354 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:41.367+0000 7ff15b59e640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff15c0b22d0 con 0x7ff15c0a9c00 2024-03-29T11:26:41.354 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:41.367+0000 7ff15ad9d640 1 --1- >> v1:172.21.15.84:6790/0 conn(0x7ff15c0b38c0 0x7ff15c14d7d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.84:6790/0 says I am v1:172.21.15.84:55726/0 (socket says 172.21.15.84:55726) 2024-03-29T11:26:41.354 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:41.367+0000 7ff15ad9d640 1 -- 172.21.15.84:0/3708814548 learned_addr learned my addr 172.21.15.84:0/3708814548 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-29T11:26:41.354 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:41.367+0000 7ff15b59e640 1 -- 172.21.15.84:0/3708814548 --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff15c142d30 con 0x7ff15c0b38c0 2024-03-29T11:26:41.354 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:41.367+0000 7ff15b59e640 10 monclient(hunting): _renew_subs 2024-03-29T11:26:41.354 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:41.367+0000 7ff15b59e640 10 monclient(hunting): authenticate will time out at 1262.661607s 2024-03-29T11:26:41.361 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:41.370+0000 7ff159d9b640 1 -- 172.21.15.84:0/3708814548 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1048041319 0 0) 0x7ff15c142d30 con 0x7ff15c0b38c0 2024-03-29T11:26:41.361 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:41.370+0000 7ff159d9b640 10 monclient(hunting): _negotiate 2024-03-29T11:26:41.361 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:41.370+0000 7ff159d9b640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:26:41.361 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:41.370+0000 7ff159d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:26:41.361 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:41.370+0000 7ff159d9b640 10 monclient(hunting): my global_id is 4511 2024-03-29T11:26:41.361 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:41.370+0000 7ff159d9b640 1 -- 172.21.15.84:0/3708814548 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff148003520 con 0x7ff15c0b38c0 2024-03-29T11:26:41.362 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:41.371+0000 7ff159d9b640 1 -- 172.21.15.84:0/3708814548 <== mon.2 v1:172.21.15.84:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1722162912 0 0) 0x7ff148003520 con 0x7ff15c0b38c0 2024-03-29T11:26:41.362 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:41.371+0000 7ff159d9b640 1 -- 172.21.15.84:0/3708814548 >> v1:172.21.15.179:6789/0 conn(0x7ff15c0a9c00 legacy=0x7ff15c13bb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:26:41.362 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:41.371+0000 7ff159d9b640 1 -- 172.21.15.84:0/3708814548 >> v1:172.21.15.84:6789/0 conn(0x7ff15c0b2800 legacy=0x7ff15c142620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:41.362 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:41.371+0000 7ff159d9b640 10 monclient: _finish_hunting 0 2024-03-29T11:26:41.362 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:41.371+0000 7ff159d9b640 1 monclient: found mon.noname-b 2024-03-29T11:26:41.362 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:41.371+0000 7ff159d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:26:41.362 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:41.371+0000 7ff159d9b640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.84:6790/0 2024-03-29T11:26:41.362 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:41.371+0000 7ff159d9b640 1 -- 172.21.15.84:0/3708814548 --> v1:172.21.15.84:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff15c0aaf90 con 0x7ff15c0b38c0 2024-03-29T11:26:41.362 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:41.371+0000 7ff159d9b640 10 monclient: _finish_auth 0 2024-03-29T11:26:41.362 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:41.371+0000 7ff159d9b640 10 monclient: _check_auth_tickets 2024-03-29T11:26:41.362 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:41.371+0000 7ff159d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:26:41.362 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:41.371+0000 7ff159d9b640 1 -- 172.21.15.84:0/3708814548 <== mon.2 v1:172.21.15.84:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4236868694 0 0) 0x7ff154001a10 con 0x7ff15c0b38c0 2024-03-29T11:26:41.362 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:41.371+0000 7ff159d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:41.362 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:41.371+0000 7ff159d9b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:26:41.362 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:41.371+0000 7ff159d9b640 10 monclient: dump: 2024-03-29T11:26:41.363 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T11:26:41.363 INFO:teuthology.misc.health.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:26:41.363 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T11:18:53.864025+0000 2024-03-29T11:26:41.363 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T11:18:53.864025+0000 2024-03-29T11:26:41.363 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:41.363 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:26:41.363 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T11:26:41.363 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T11:26:41.363 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T11:26:41.363 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:26:41.363 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:41.371+0000 7ff159d9b640 1 -- 172.21.15.84:0/3708814548 <== mon.2 v1:172.21.15.84:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff154001c30 con 0x7ff15c0b38c0 2024-03-29T11:26:41.363 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:41.371+0000 7ff159d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:26:41.363 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:41.371+0000 7ff159d9b640 1 -- 172.21.15.84:0/3708814548 <== mon.2 v1:172.21.15.84:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4236868694 0 0) 0x7ff154001fc0 con 0x7ff15c0b38c0 2024-03-29T11:26:41.363 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:41.371+0000 7ff159d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:41.363 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:41.371+0000 7ff159d9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:26:41.364 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:41.371+0000 7ff159d9b640 10 monclient: dump: 2024-03-29T11:26:41.364 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T11:26:41.364 INFO:teuthology.misc.health.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:26:41.364 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T11:18:53.864025+0000 2024-03-29T11:26:41.364 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T11:18:53.864025+0000 2024-03-29T11:26:41.364 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:41.364 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:26:41.364 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T11:26:41.364 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T11:26:41.364 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T11:26:41.364 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:26:41.364 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:41.372+0000 7ff15b59e640 5 monclient: authenticate success, global_id 4511 2024-03-29T11:26:41.364 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:41.372+0000 7ff15b59e640 10 monclient: get_monmap_and_config success 2024-03-29T11:26:41.364 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:41.372+0000 7ff15b59e640 4 set_mon_vals no callback set 2024-03-29T11:26:41.364 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:41.373+0000 7ff15b59e640 10 monclient: shutdown 2024-03-29T11:26:41.365 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:41.373+0000 7ff15b59e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:26:41.365 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:41.373+0000 7ff15b59e640 1 -- 172.21.15.84:0/3708814548 >> v1:172.21.15.84:6790/0 conn(0x7ff15c0b38c0 legacy=0x7ff15c14d7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:41.365 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:41.374+0000 7ff15b59e640 1 -- 172.21.15.84:0/3708814548 shutdown_connections 2024-03-29T11:26:41.365 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:41.374+0000 7ff15b59e640 1 -- 172.21.15.84:0/3708814548 >> 172.21.15.84:0/3708814548 conn(0x7ff15c066a00 msgr2=0x7ff15c066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:26:41.365 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:41.374+0000 7ff15b59e640 1 -- 172.21.15.84:0/3708814548 shutdown_connections 2024-03-29T11:26:41.365 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:41.374+0000 7ff15b59e640 1 -- 172.21.15.84:0/3708814548 wait complete. 2024-03-29T11:26:41.366 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:41.375+0000 7ff15b59e640 10 monclient: build_initial_monmap 2024-03-29T11:26:41.366 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:41.375+0000 7ff15b59e640 10 monclient: monmap: 2024-03-29T11:26:41.366 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-29T11:26:41.366 INFO:teuthology.misc.health.smithi084.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:26:41.366 INFO:teuthology.misc.health.smithi084.stderr:last_changed 0.000000 2024-03-29T11:26:41.366 INFO:teuthology.misc.health.smithi084.stderr:created 0.000000 2024-03-29T11:26:41.366 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-29T11:26:41.366 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:26:41.366 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-29T11:26:41.366 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.noname-b 2024-03-29T11:26:41.366 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-c 2024-03-29T11:26:41.367 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:26:41.367 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:41.376+0000 7ff15b59e640 1 librados: starting msgr at 2024-03-29T11:26:41.367 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:41.376+0000 7ff15b59e640 1 librados: starting objecter 2024-03-29T11:26:41.367 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:41.376+0000 7ff15b59e640 1 Processor -- start 2024-03-29T11:26:41.367 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:41.377+0000 7ff15b59e640 1 -- start start 2024-03-29T11:26:41.367 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:41.377+0000 7ff15b59e640 1 librados: setting wanted keys 2024-03-29T11:26:41.367 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:41.377+0000 7ff15b59e640 1 librados: calling monclient init 2024-03-29T11:26:41.367 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:41.377+0000 7ff15b59e640 10 monclient: init 2024-03-29T11:26:41.367 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:41.377+0000 7ff15b59e640 10 monclient: _reopen_session rank -1 2024-03-29T11:26:41.367 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:41.377+0000 7ff15b59e640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:26:41.369 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:41.381+0000 7ff15b59e640 10 monclient(hunting): picked mon.noname-a con 0x7ff15c0a9c00 addr v1:172.21.15.84:6789/0 2024-03-29T11:26:41.370 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:41.381+0000 7ff15b59e640 10 monclient(hunting): picked mon.noname-c con 0x7ff15c0b2800 addr v1:172.21.15.84:6790/0 2024-03-29T11:26:41.370 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:41.381+0000 7ff15b59e640 10 monclient(hunting): picked mon.noname-b con 0x7ff15c0b38c0 addr v1:172.21.15.179:6789/0 2024-03-29T11:26:41.370 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:41.381+0000 7ff15b59e640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff15c13f610 con 0x7ff15c0a9c00 2024-03-29T11:26:41.370 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:41.381+0000 7ff15b59e640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff15c13c8d0 con 0x7ff15c0b38c0 2024-03-29T11:26:41.370 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:41.381+0000 7ff15b59e640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff15c0ab370 con 0x7ff15c0b2800 2024-03-29T11:26:41.370 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:41.381+0000 7ff15b59e640 10 monclient(hunting): _renew_subs 2024-03-29T11:26:41.370 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:41.381+0000 7ff15b59e640 10 monclient(hunting): authenticate will time out at 1262.675394s 2024-03-29T11:26:41.370 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:41.381+0000 7ff151d9b640 1 --1- >> v1:172.21.15.84:6790/0 conn(0x7ff15c0b2800 0x7ff15c1da380 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.84:6790/0 says I am v1:172.21.15.84:56212/0 (socket says 172.21.15.84:56212) 2024-03-29T11:26:41.370 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:41.381+0000 7ff151d9b640 1 -- 172.21.15.84:0/2558743909 learned_addr learned my addr 172.21.15.84:0/2558743909 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-29T11:26:41.370 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:41.382+0000 7ff1537fe640 10 client.?.objecter ms_handle_connect 0x7ff15c0a9c00 2024-03-29T11:26:41.370 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:41.382+0000 7ff1537fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:26:41.370 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:41.382+0000 7ff1537fe640 10 client.?.objecter ms_handle_connect 0x7ff15c0b2800 2024-03-29T11:26:41.370 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:41.382+0000 7ff1537fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:26:41.371 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:41.382+0000 7ff1537fe640 1 -- 172.21.15.84:0/2558743909 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2032651196 0 0) 0x7ff15c0ab370 con 0x7ff15c0b2800 2024-03-29T11:26:41.371 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:41.382+0000 7ff1537fe640 10 monclient(hunting): _negotiate 2024-03-29T11:26:41.371 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:41.382+0000 7ff1537fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:26:41.371 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:41.382+0000 7ff1537fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:26:41.371 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:41.382+0000 7ff1537fe640 10 monclient(hunting): my global_id is 4514 2024-03-29T11:26:41.371 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:41.382+0000 7ff1537fe640 1 -- 172.21.15.84:0/2558743909 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff13c004eb0 con 0x7ff15c0b2800 2024-03-29T11:26:41.371 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:41.382+0000 7ff1537fe640 1 -- 172.21.15.84:0/2558743909 <== mon.2 v1:172.21.15.84:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1162215832 0 0) 0x7ff13c004eb0 con 0x7ff15c0b2800 2024-03-29T11:26:41.371 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:41.382+0000 7ff1537fe640 1 -- 172.21.15.84:0/2558743909 --> v1:172.21.15.84:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff15c0ab370 con 0x7ff15c0b2800 2024-03-29T11:26:41.371 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:41.382+0000 7ff1537fe640 1 -- 172.21.15.84:0/2558743909 <== mon.2 v1:172.21.15.84:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4236868694 0 0) 0x7ff144003f80 con 0x7ff15c0b2800 2024-03-29T11:26:41.371 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:41.382+0000 7ff1537fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:41.371 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:41.382+0000 7ff1537fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:26:41.371 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:41.382+0000 7ff1537fe640 10 monclient(hunting): dump: 2024-03-29T11:26:41.371 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T11:26:41.371 INFO:teuthology.misc.health.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:26:41.372 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T11:18:53.864025+0000 2024-03-29T11:26:41.372 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T11:18:53.864025+0000 2024-03-29T11:26:41.372 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:41.372 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:26:41.372 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T11:26:41.372 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T11:26:41.372 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T11:26:41.372 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:26:41.372 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:41.382+0000 7ff1537fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:26:41.372 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:41.383+0000 7ff1537fe640 1 -- 172.21.15.84:0/2558743909 <== mon.0 v1:172.21.15.84:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1449781613 0 0) 0x7ff15c13f610 con 0x7ff15c0a9c00 2024-03-29T11:26:41.372 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:41.383+0000 7ff1537fe640 10 monclient(hunting): _negotiate 2024-03-29T11:26:41.372 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:41.383+0000 7ff1537fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:26:41.372 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:41.383+0000 7ff1537fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:26:41.372 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:41.383+0000 7ff1537fe640 10 monclient(hunting): my global_id is 4491 2024-03-29T11:26:41.372 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:41.383+0000 7ff1537fe640 1 -- 172.21.15.84:0/2558743909 --> v1:172.21.15.84:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff13c004eb0 con 0x7ff15c0a9c00 2024-03-29T11:26:41.373 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:41.383+0000 7ff1537fe640 1 -- 172.21.15.84:0/2558743909 <== mon.2 v1:172.21.15.84:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2433580461 0 0) 0x7ff15c0ab370 con 0x7ff15c0b2800 2024-03-29T11:26:41.373 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:41.383+0000 7ff1537fe640 1 -- 172.21.15.84:0/2558743909 >> v1:172.21.15.179:6789/0 conn(0x7ff15c0b38c0 legacy=0x7ff15c1daa90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:26:41.376 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:41.384+0000 7ff1537fe640 1 -- 172.21.15.84:0/2558743909 >> v1:172.21.15.84:6789/0 conn(0x7ff15c0a9c00 legacy=0x7ff15c0b0be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:41.376 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:41.384+0000 7ff1537fe640 10 monclient: _finish_hunting 0 2024-03-29T11:26:41.376 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:41.384+0000 7ff1537fe640 1 monclient: found mon.c 2024-03-29T11:26:41.376 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:41.384+0000 7ff1537fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:26:41.376 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:41.384+0000 7ff1537fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-29T11:26:41.376 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:41.384+0000 7ff1537fe640 1 -- 172.21.15.84:0/2558743909 --> v1:172.21.15.84:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff15c13f1a0 con 0x7ff15c0b2800 2024-03-29T11:26:41.376 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:41.384+0000 7ff1537fe640 10 monclient: _finish_auth 0 2024-03-29T11:26:41.376 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:41.384+0000 7ff1537fe640 10 monclient: _check_auth_tickets 2024-03-29T11:26:41.376 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:41.384+0000 7ff1537fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:26:41.376 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:41.384+0000 7ff15b59e640 5 monclient: authenticate success, global_id 4514 2024-03-29T11:26:41.376 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:41.384+0000 7ff15b59e640 10 monclient: _renew_subs 2024-03-29T11:26:41.377 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:41.384+0000 7ff15b59e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-29T11:26:41.377 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:41.384+0000 7ff15b59e640 1 -- 172.21.15.84:0/2558743909 --> v1:172.21.15.84:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff15c0abb60 con 0x7ff15c0b2800 2024-03-29T11:26:41.377 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:41.384+0000 7ff15b59e640 10 client.4514.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:26:41.377 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:41.384+0000 7ff15b59e640 10 monclient: _renew_subs 2024-03-29T11:26:41.377 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:41.384+0000 7ff15b59e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-29T11:26:41.377 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:41.384+0000 7ff15b59e640 1 -- 172.21.15.84:0/2558743909 --> v1:172.21.15.84:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff15c0ac1b0 con 0x7ff15c0b2800 2024-03-29T11:26:41.377 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:41.384+0000 7ff15b59e640 1 librados: init done 2024-03-29T11:26:41.377 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:41.386+0000 7ff1537fe640 1 -- 172.21.15.84:0/2558743909 <== mon.2 v1:172.21.15.84:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff144000ed0 con 0x7ff15c0b2800 2024-03-29T11:26:41.377 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:41.386+0000 7ff1537fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:26:41.377 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:41.386+0000 7ff1537fe640 1 -- 172.21.15.84:0/2558743909 <== mon.2 v1:172.21.15.84:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4236868694 0 0) 0x7ff1440040e0 con 0x7ff15c0b2800 2024-03-29T11:26:41.377 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:41.386+0000 7ff1537fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:41.377 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:41.386+0000 7ff1537fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:26:41.377 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:41.386+0000 7ff1537fe640 10 monclient: dump: 2024-03-29T11:26:41.377 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T11:26:41.378 INFO:teuthology.misc.health.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:26:41.378 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T11:18:53.864025+0000 2024-03-29T11:26:41.378 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T11:18:53.864025+0000 2024-03-29T11:26:41.378 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:41.378 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:26:41.378 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T11:26:41.378 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T11:26:41.378 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T11:26:41.378 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:26:41.378 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:41.386+0000 7ff15b59e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:26:41.378 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:41.386+0000 7ff15b59e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:26:41.378 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:41.386+0000 7ff15b59e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-29T11:26:41.378 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:41.386+0000 7ff15b59e640 1 -- 172.21.15.84:0/2558743909 --> v1:172.21.15.84:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff15c0ad160 con 0x7ff15c0b2800 2024-03-29T11:26:41.378 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:41.387+0000 7ff158d99640 4 set_mon_vals no callback set 2024-03-29T11:26:41.379 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:41.389+0000 7ff1537fe640 1 -- 172.21.15.84:0/2558743909 <== mon.2 v1:172.21.15.84:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4033942909 0 0) 0x7ff1440012a0 con 0x7ff15c0b2800 2024-03-29T11:26:41.379 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:41.390+0000 7ff1537fe640 1 -- 172.21.15.84:0/2558743909 <== mon.2 v1:172.21.15.84:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2808139587 0 0) 0x7ff144084a00 con 0x7ff15c0b2800 2024-03-29T11:26:41.380 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:41.390+0000 7ff1537fe640 10 client.4514.objecter ms_dispatch 0x7ff15c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:26:41.380 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:41.390+0000 7ff1537fe640 3 client.4514.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:26:41.380 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:41.390+0000 7ff1537fe640 3 client.4514.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:26:41.380 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:41.390+0000 7ff1537fe640 20 client.4514.objecter dump_active .. 0 homeless 2024-03-29T11:26:41.380 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:41.390+0000 7ff15a59c640 10 monclient: get_auth_request con 0x7ff13c06dbd0 auth_method 0 2024-03-29T11:26:41.380 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:41.391+0000 7ff1537fe640 10 client.4514.objecter ms_handle_connect 0x7ff13c06dbd0 2024-03-29T11:26:41.384 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:41.397+0000 7ff1537fe640 1 -- 172.21.15.84:0/2558743909 <== mon.2 v1:172.21.15.84:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7ff1440502e0 con 0x7ff15c0b2800 2024-03-29T11:26:41.384 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:41.397+0000 7ff1537fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:26:41.384 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:41.397+0000 7ff1537fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:26:41.716 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:41.729+0000 7ff15b59e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:26:41.716 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:41.729+0000 7ff15b59e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:26:41.716 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:41.729+0000 7ff15b59e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-29T11:26:41.716 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:41.729+0000 7ff15b59e640 1 -- 172.21.15.84:0/2558743909 --> v1:172.21.15.84:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff15c04f730 con 0x7ff15c0b2800 2024-03-29T11:26:41.717 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:41.731+0000 7ff1537fe640 1 -- 172.21.15.84:0/2558743909 <== mon.2 v1:172.21.15.84:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4209477549) 0x7ff144052210 con 0x7ff15c0b2800 2024-03-29T11:26:41.717 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:41.731+0000 7ff1537fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:26:41.718 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:41.731+0000 7ff1537fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:26:41.718 INFO:teuthology.misc.health.smithi084.stdout: 2024-03-29T11:26:41.718 INFO:teuthology.misc.health.smithi084.stdout:{"epoch":10,"fsid":"2c3e002f-9eb0-4860-853b-4ef1ef562213","created":"2024-03-29T11:19:32.029169+0000","modified":"2024-03-29T11:19:34.200299+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:34.200299+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"5c39e276-2034-4716-bbc0-57a634eefa92","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"cd580e99-109f-4bf1-88c0-a5987bc8b9ed","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"11e60a5a-051d-453f-a190-66387cb35592","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"2c329fbe-bbc4-4030-bf87-d711cad04679","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f4213d4a-bd36-4433-91ae-d7e2b02481d3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"f0cf44ea-6055-48af-805b-aed858102ff9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"d895eaa0-cc0b-4ee4-9b5a-2dfcd89a60b7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"9bc5a7bc-8f52-4db9-96d6-de5e68d52eaa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:26:41.721 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:41.735+0000 7ff15b59e640 10 librados: watch_flush enter 2024-03-29T11:26:41.721 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:41.735+0000 7ff15b59e640 10 librados: watch_flush exit 2024-03-29T11:26:41.721 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:41.735+0000 7ff15b59e640 20 client.4514.objecter shutdown clearing up homeless session... 2024-03-29T11:26:41.721 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:41.735+0000 7ff15b59e640 10 client.4514.objecter successfully canceled tick 2024-03-29T11:26:41.722 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:41.735+0000 7ff15b59e640 1 -- 172.21.15.84:0/2558743909 >> v1:172.21.15.179:6816/2981134980 conn(0x7ff13c06dbd0 legacy=0x7ff13c070090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:41.722 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:41.735+0000 7ff15b59e640 10 monclient: shutdown 2024-03-29T11:26:41.722 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:41.735+0000 7ff15b59e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:26:41.722 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:41.735+0000 7ff15b59e640 1 -- 172.21.15.84:0/2558743909 >> v1:172.21.15.84:6790/0 conn(0x7ff15c0b2800 legacy=0x7ff15c1da380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:41.722 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:41.735+0000 7ff15b59e640 1 -- 172.21.15.84:0/2558743909 shutdown_connections 2024-03-29T11:26:41.722 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:41.735+0000 7ff15b59e640 1 -- 172.21.15.84:0/2558743909 >> 172.21.15.84:0/2558743909 conn(0x7ff15c066a00 msgr2=0x7ff15c13bb00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:26:41.723 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:41.736+0000 7ff15b59e640 1 -- 172.21.15.84:0/2558743909 shutdown_connections 2024-03-29T11:26:41.723 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:41.736+0000 7ff15b59e640 1 -- 172.21.15.84:0/2558743909 wait complete. 2024-03-29T11:26:41.723 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:41.736+0000 7ff15b59e640 1 librados: shutdown 2024-03-29T11:26:41.739 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:26:48.546 DEBUG:teuthology.orchestra.run.smithi084:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:26:48.695 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:48.708+0000 7f766b59e640 10 monclient: get_monmap_and_config 2024-03-29T11:26:48.695 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:48.708+0000 7f766b59e640 10 monclient: build_initial_monmap 2024-03-29T11:26:48.695 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:48.708+0000 7f766b59e640 10 monclient: monmap: 2024-03-29T11:26:48.695 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-29T11:26:48.695 INFO:teuthology.misc.health.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:26:48.695 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T11:26:48.709042+0000 2024-03-29T11:26:48.695 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T11:26:48.709042+0000 2024-03-29T11:26:48.695 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-29T11:26:48.695 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:26:48.695 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-29T11:26:48.696 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.noname-c 2024-03-29T11:26:48.696 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-b 2024-03-29T11:26:48.696 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:26:48.698 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:48.709+0000 7f766b59e640 1 Processor -- start 2024-03-29T11:26:48.698 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:48.709+0000 7f766b59e640 1 -- start start 2024-03-29T11:26:48.698 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:48.709+0000 7f766b59e640 10 monclient: init 2024-03-29T11:26:48.698 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:48.710+0000 7f766b59e640 10 monclient: _reopen_session rank -1 2024-03-29T11:26:48.698 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:48.710+0000 7f766b59e640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:26:48.698 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:48.710+0000 7f766b59e640 10 monclient(hunting): picked mon.noname-b con 0x7f76640cf4d0 addr v1:172.21.15.84:6790/0 2024-03-29T11:26:48.698 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:48.710+0000 7f766b59e640 10 monclient(hunting): picked mon.noname-c con 0x7f7664031eb0 addr v1:172.21.15.179:6789/0 2024-03-29T11:26:48.699 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:48.710+0000 7f766b59e640 10 monclient(hunting): picked mon.noname-a con 0x7f7664036dc0 addr v1:172.21.15.84:6789/0 2024-03-29T11:26:48.699 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:48.710+0000 7f766b59e640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f76640336a0 con 0x7f7664036dc0 2024-03-29T11:26:48.699 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:48.710+0000 7f766b59e640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7664030860 con 0x7f7664031eb0 2024-03-29T11:26:48.699 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:48.710+0000 7f766b59e640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f76640338f0 con 0x7f76640cf4d0 2024-03-29T11:26:48.699 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:48.710+0000 7f766b59e640 10 monclient(hunting): _renew_subs 2024-03-29T11:26:48.699 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:48.710+0000 7f766b59e640 10 monclient(hunting): authenticate will time out at 1270.005022s 2024-03-29T11:26:48.700 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:48.713+0000 7f766a59c640 1 --1- >> v1:172.21.15.84:6790/0 conn(0x7f76640cf4d0 0x7f76640cfbe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.84:6790/0 says I am v1:172.21.15.84:53614/0 (socket says 172.21.15.84:53614) 2024-03-29T11:26:48.701 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:48.713+0000 7f766a59c640 1 -- 172.21.15.84:0/157219409 learned_addr learned my addr 172.21.15.84:0/157219409 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-29T11:26:48.704 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:48.716+0000 7f766959a640 1 -- 172.21.15.84:0/157219409 <== mon.0 v1:172.21.15.84:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3736323082 0 0) 0x7f76640336a0 con 0x7f7664036dc0 2024-03-29T11:26:48.704 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:48.716+0000 7f766959a640 10 monclient(hunting): _negotiate 2024-03-29T11:26:48.704 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:48.716+0000 7f766959a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:26:48.704 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:48.716+0000 7f766959a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:26:48.704 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:48.716+0000 7f766959a640 10 monclient(hunting): my global_id is 4494 2024-03-29T11:26:48.704 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:48.716+0000 7f766959a640 1 -- 172.21.15.84:0/157219409 --> v1:172.21.15.84:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7654003520 con 0x7f7664036dc0 2024-03-29T11:26:48.704 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:48.716+0000 7f766959a640 1 -- 172.21.15.84:0/157219409 <== mon.0 v1:172.21.15.84:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2528625891 0 0) 0x7f7654003520 con 0x7f7664036dc0 2024-03-29T11:26:48.704 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:48.716+0000 7f766959a640 1 -- 172.21.15.84:0/157219409 >> v1:172.21.15.84:6790/0 conn(0x7f76640cf4d0 legacy=0x7f76640cfbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:48.704 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:48.716+0000 7f766959a640 1 -- 172.21.15.84:0/157219409 >> v1:172.21.15.179:6789/0 conn(0x7f7664031eb0 legacy=0x7f76640366b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:48.704 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:48.716+0000 7f766959a640 10 monclient: _finish_hunting 0 2024-03-29T11:26:48.705 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:48.716+0000 7f766959a640 1 monclient: found mon.noname-a 2024-03-29T11:26:48.705 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:48.716+0000 7f766959a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:26:48.705 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:48.716+0000 7f766959a640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.84:6789/0 2024-03-29T11:26:48.705 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:48.716+0000 7f766959a640 1 -- 172.21.15.84:0/157219409 --> v1:172.21.15.84:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7664033bd0 con 0x7f7664036dc0 2024-03-29T11:26:48.705 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:48.716+0000 7f766959a640 10 monclient: _finish_auth 0 2024-03-29T11:26:48.705 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:48.716+0000 7f766959a640 10 monclient: _check_auth_tickets 2024-03-29T11:26:48.705 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:48.716+0000 7f766959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:26:48.705 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:48.716+0000 7f766959a640 1 -- 172.21.15.84:0/157219409 <== mon.0 v1:172.21.15.84:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4236868694 0 0) 0x7f7660001690 con 0x7f7664036dc0 2024-03-29T11:26:48.705 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:48.716+0000 7f766959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:48.705 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:48.716+0000 7f766959a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:26:48.705 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:48.716+0000 7f766959a640 10 monclient: dump: 2024-03-29T11:26:48.705 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T11:26:48.705 INFO:teuthology.misc.health.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:26:48.705 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T11:18:53.864025+0000 2024-03-29T11:26:48.706 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T11:18:53.864025+0000 2024-03-29T11:26:48.706 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:48.706 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:26:48.706 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T11:26:48.706 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T11:26:48.706 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T11:26:48.706 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:26:48.706 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:48.716+0000 7f766b59e640 5 monclient: authenticate success, global_id 4494 2024-03-29T11:26:48.706 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:48.716+0000 7f766b59e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:26:48.706 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:48.717+0000 7f766959a640 1 -- 172.21.15.84:0/157219409 <== mon.0 v1:172.21.15.84:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f76600017f0 con 0x7f7664036dc0 2024-03-29T11:26:48.706 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:48.717+0000 7f766959a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:26:48.706 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:48.717+0000 7f766959a640 1 -- 172.21.15.84:0/157219409 <== mon.0 v1:172.21.15.84:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4236868694 0 0) 0x7f7660001c00 con 0x7f7664036dc0 2024-03-29T11:26:48.706 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:48.717+0000 7f766959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:48.707 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:48.717+0000 7f766959a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:26:48.707 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:48.717+0000 7f766959a640 10 monclient: dump: 2024-03-29T11:26:48.707 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T11:26:48.707 INFO:teuthology.misc.health.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:26:48.707 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T11:18:53.864025+0000 2024-03-29T11:26:48.707 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T11:18:53.864025+0000 2024-03-29T11:26:48.707 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:48.707 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:26:48.707 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T11:26:48.707 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T11:26:48.707 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T11:26:48.707 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:26:48.707 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:48.717+0000 7f766b59e640 10 monclient: get_monmap_and_config success 2024-03-29T11:26:48.707 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:48.717+0000 7f766b59e640 4 set_mon_vals no callback set 2024-03-29T11:26:48.710 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:48.718+0000 7f766b59e640 10 monclient: shutdown 2024-03-29T11:26:48.710 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:48.718+0000 7f766b59e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:26:48.711 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:48.718+0000 7f766b59e640 1 -- 172.21.15.84:0/157219409 >> v1:172.21.15.84:6789/0 conn(0x7f7664036dc0 legacy=0x7f76640d7600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:48.711 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:48.718+0000 7f766b59e640 1 -- 172.21.15.84:0/157219409 shutdown_connections 2024-03-29T11:26:48.711 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:48.718+0000 7f766b59e640 1 -- 172.21.15.84:0/157219409 >> 172.21.15.84:0/157219409 conn(0x7f766403d480 msgr2=0x7f766403d890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:26:48.711 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:48.718+0000 7f766b59e640 1 -- 172.21.15.84:0/157219409 shutdown_connections 2024-03-29T11:26:48.711 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:48.718+0000 7f766b59e640 1 -- 172.21.15.84:0/157219409 wait complete. 2024-03-29T11:26:48.711 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:48.720+0000 7f766b59e640 10 monclient: build_initial_monmap 2024-03-29T11:26:48.711 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:48.720+0000 7f766b59e640 10 monclient: monmap: 2024-03-29T11:26:48.711 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-29T11:26:48.711 INFO:teuthology.misc.health.smithi084.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:26:48.711 INFO:teuthology.misc.health.smithi084.stderr:last_changed 0.000000 2024-03-29T11:26:48.711 INFO:teuthology.misc.health.smithi084.stderr:created 0.000000 2024-03-29T11:26:48.711 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-29T11:26:48.711 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:26:48.712 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-29T11:26:48.712 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.noname-b 2024-03-29T11:26:48.712 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-c 2024-03-29T11:26:48.712 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:26:48.712 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:48.720+0000 7f766b59e640 1 librados: starting msgr at 2024-03-29T11:26:48.712 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:48.720+0000 7f766b59e640 1 librados: starting objecter 2024-03-29T11:26:48.712 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:48.720+0000 7f766b59e640 1 Processor -- start 2024-03-29T11:26:48.712 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:48.720+0000 7f766b59e640 1 -- start start 2024-03-29T11:26:48.712 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:48.720+0000 7f766b59e640 1 librados: setting wanted keys 2024-03-29T11:26:48.712 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:48.720+0000 7f766b59e640 1 librados: calling monclient init 2024-03-29T11:26:48.712 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:48.720+0000 7f766b59e640 10 monclient: init 2024-03-29T11:26:48.712 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:48.721+0000 7f766b59e640 10 monclient: _reopen_session rank -1 2024-03-29T11:26:48.712 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:48.721+0000 7f766b59e640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:26:48.713 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:48.721+0000 7f766b59e640 10 monclient(hunting): picked mon.noname-b con 0x7f7664031eb0 addr v1:172.21.15.179:6789/0 2024-03-29T11:26:48.713 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:48.721+0000 7f766b59e640 10 monclient(hunting): picked mon.noname-c con 0x7f7664036dc0 addr v1:172.21.15.84:6790/0 2024-03-29T11:26:48.713 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:48.721+0000 7f766b59e640 10 monclient(hunting): picked mon.noname-a con 0x7f76640cf4d0 addr v1:172.21.15.84:6789/0 2024-03-29T11:26:48.713 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:48.721+0000 7f766b59e640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f76640cc970 con 0x7f76640cf4d0 2024-03-29T11:26:48.713 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:48.721+0000 7f766b59e640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f76640ca7d0 con 0x7f7664031eb0 2024-03-29T11:26:48.713 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:48.721+0000 7f766b59e640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f766402e6f0 con 0x7f7664036dc0 2024-03-29T11:26:48.713 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:48.721+0000 7f766b59e640 10 monclient(hunting): _renew_subs 2024-03-29T11:26:48.713 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:48.721+0000 7f766b59e640 10 monclient(hunting): authenticate will time out at 1270.015845s 2024-03-29T11:26:48.713 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:48.722+0000 7f7669d9b640 1 --1- >> v1:172.21.15.84:6790/0 conn(0x7f7664036dc0 0x7f766402ec10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.84:6790/0 says I am v1:172.21.15.84:55242/0 (socket says 172.21.15.84:55242) 2024-03-29T11:26:48.713 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:48.722+0000 7f7669d9b640 1 -- 172.21.15.84:0/2737032010 learned_addr learned my addr 172.21.15.84:0/2737032010 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-29T11:26:48.721 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:48.726+0000 7f765affd640 10 client.?.objecter ms_handle_connect 0x7f7664036dc0 2024-03-29T11:26:48.721 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:48.726+0000 7f765affd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:26:48.721 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:48.726+0000 7f765affd640 10 client.?.objecter ms_handle_connect 0x7f7664031eb0 2024-03-29T11:26:48.721 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:48.726+0000 7f765affd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:26:48.721 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:48.726+0000 7f765affd640 10 client.?.objecter ms_handle_connect 0x7f76640cf4d0 2024-03-29T11:26:48.721 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:48.726+0000 7f765affd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:26:48.721 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:48.726+0000 7f765affd640 1 -- 172.21.15.84:0/2737032010 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 868385957 0 0) 0x7f766402e6f0 con 0x7f7664036dc0 2024-03-29T11:26:48.721 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:48.726+0000 7f765affd640 10 monclient(hunting): _negotiate 2024-03-29T11:26:48.721 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:48.726+0000 7f765affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:26:48.722 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:48.726+0000 7f765affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:26:48.722 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:48.726+0000 7f765affd640 10 monclient(hunting): my global_id is 4520 2024-03-29T11:26:48.722 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:48.726+0000 7f765affd640 1 -- 172.21.15.84:0/2737032010 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f764c004eb0 con 0x7f7664036dc0 2024-03-29T11:26:48.722 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:48.726+0000 7f765affd640 1 -- 172.21.15.84:0/2737032010 <== mon.0 v1:172.21.15.84:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2645221032 0 0) 0x7f76640cc970 con 0x7f76640cf4d0 2024-03-29T11:26:48.722 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:48.727+0000 7f765affd640 10 monclient(hunting): _negotiate 2024-03-29T11:26:48.722 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:48.727+0000 7f765affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:26:48.722 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:48.727+0000 7f765affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:26:48.722 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:48.727+0000 7f765affd640 10 monclient(hunting): my global_id is 4497 2024-03-29T11:26:48.722 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:48.727+0000 7f765affd640 1 -- 172.21.15.84:0/2737032010 --> v1:172.21.15.84:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f766402e6f0 con 0x7f76640cf4d0 2024-03-29T11:26:48.722 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:48.727+0000 7f765affd640 1 -- 172.21.15.84:0/2737032010 <== mon.1 v1:172.21.15.179:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1087134848 0 0) 0x7f76640ca7d0 con 0x7f7664031eb0 2024-03-29T11:26:48.722 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:48.727+0000 7f765affd640 10 monclient(hunting): _negotiate 2024-03-29T11:26:48.722 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:48.727+0000 7f765affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:26:48.722 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:48.727+0000 7f765affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:26:48.723 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:48.727+0000 7f765affd640 10 monclient(hunting): my global_id is 4441 2024-03-29T11:26:48.723 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:48.727+0000 7f765affd640 1 -- 172.21.15.84:0/2737032010 --> v1:172.21.15.179:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f76640cc970 con 0x7f7664031eb0 2024-03-29T11:26:48.723 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:48.727+0000 7f765affd640 1 -- 172.21.15.84:0/2737032010 <== mon.2 v1:172.21.15.84:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1527815246 0 0) 0x7f764c004eb0 con 0x7f7664036dc0 2024-03-29T11:26:48.723 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:48.727+0000 7f765affd640 1 -- 172.21.15.84:0/2737032010 --> v1:172.21.15.84:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f76640ca7d0 con 0x7f7664036dc0 2024-03-29T11:26:48.723 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:48.727+0000 7f765affd640 1 -- 172.21.15.84:0/2737032010 <== mon.0 v1:172.21.15.84:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3876777623 0 0) 0x7f766402e6f0 con 0x7f76640cf4d0 2024-03-29T11:26:48.723 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:48.727+0000 7f765affd640 1 -- 172.21.15.84:0/2737032010 --> v1:172.21.15.84:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f764c004eb0 con 0x7f76640cf4d0 2024-03-29T11:26:48.723 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:48.728+0000 7f765affd640 1 -- 172.21.15.84:0/2737032010 <== mon.2 v1:172.21.15.84:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4236868694 0 0) 0x7f765c0019b0 con 0x7f7664036dc0 2024-03-29T11:26:48.723 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:48.728+0000 7f765affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:48.723 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:48.728+0000 7f765affd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:26:48.723 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:48.728+0000 7f765affd640 10 monclient(hunting): dump: 2024-03-29T11:26:48.723 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T11:26:48.723 INFO:teuthology.misc.health.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:26:48.723 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T11:18:53.864025+0000 2024-03-29T11:26:48.723 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T11:18:53.864025+0000 2024-03-29T11:26:48.724 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:48.724 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:26:48.724 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T11:26:48.724 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T11:26:48.724 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T11:26:48.724 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:26:48.724 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:48.728+0000 7f765affd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:26:48.724 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:48.728+0000 7f765affd640 1 -- 172.21.15.84:0/2737032010 <== mon.0 v1:172.21.15.84:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4236868694 0 0) 0x7f7660001960 con 0x7f76640cf4d0 2024-03-29T11:26:48.724 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:48.728+0000 7f765affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:48.724 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:48.728+0000 7f765affd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:26:48.724 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:48.728+0000 7f765affd640 10 monclient(hunting): dump: 2024-03-29T11:26:48.724 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T11:26:48.724 INFO:teuthology.misc.health.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:26:48.724 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T11:18:53.864025+0000 2024-03-29T11:26:48.724 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T11:18:53.864025+0000 2024-03-29T11:26:48.725 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:48.725 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:26:48.725 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T11:26:48.725 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T11:26:48.725 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T11:26:48.725 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:26:48.725 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:48.728+0000 7f765affd640 1 -- 172.21.15.84:0/2737032010 <== mon.1 v1:172.21.15.179:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1282705752 0 0) 0x7f76640cc970 con 0x7f7664031eb0 2024-03-29T11:26:48.725 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:48.728+0000 7f765affd640 1 -- 172.21.15.84:0/2737032010 --> v1:172.21.15.179:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f766402e6f0 con 0x7f7664031eb0 2024-03-29T11:26:48.725 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:48.728+0000 7f765affd640 1 -- 172.21.15.84:0/2737032010 <== mon.2 v1:172.21.15.84:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2519272264 0 0) 0x7f76640ca7d0 con 0x7f7664036dc0 2024-03-29T11:26:48.725 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:48.728+0000 7f765affd640 1 -- 172.21.15.84:0/2737032010 >> v1:172.21.15.179:6789/0 conn(0x7f7664031eb0 legacy=0x7f76640c5c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:48.725 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:48.728+0000 7f765affd640 1 -- 172.21.15.84:0/2737032010 >> v1:172.21.15.84:6789/0 conn(0x7f76640cf4d0 legacy=0x7f766402f320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:48.725 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:48.728+0000 7f765affd640 10 monclient: _finish_hunting 0 2024-03-29T11:26:48.726 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:48.728+0000 7f765affd640 1 monclient: found mon.c 2024-03-29T11:26:48.726 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:48.728+0000 7f765affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:26:48.726 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:48.728+0000 7f765affd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-29T11:26:48.726 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:48.728+0000 7f765affd640 1 -- 172.21.15.84:0/2737032010 --> v1:172.21.15.84:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76640cbc30 con 0x7f7664036dc0 2024-03-29T11:26:48.726 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:48.728+0000 7f765affd640 10 monclient: _finish_auth 0 2024-03-29T11:26:48.726 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:48.728+0000 7f765affd640 10 monclient: _check_auth_tickets 2024-03-29T11:26:48.726 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:48.728+0000 7f765affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:26:48.726 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:48.728+0000 7f766b59e640 5 monclient: authenticate success, global_id 4520 2024-03-29T11:26:48.726 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:48.728+0000 7f766b59e640 10 monclient: _renew_subs 2024-03-29T11:26:48.726 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:48.728+0000 7f766b59e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-29T11:26:48.726 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:48.728+0000 7f766b59e640 1 -- 172.21.15.84:0/2737032010 --> v1:172.21.15.84:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f76640cbe80 con 0x7f7664036dc0 2024-03-29T11:26:48.726 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:48.728+0000 7f766b59e640 10 client.4520.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:26:48.727 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:48.728+0000 7f766b59e640 10 monclient: _renew_subs 2024-03-29T11:26:48.727 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:48.728+0000 7f766b59e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-29T11:26:48.727 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:48.728+0000 7f766b59e640 1 -- 172.21.15.84:0/2737032010 --> v1:172.21.15.84:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f7664165460 con 0x7f7664036dc0 2024-03-29T11:26:48.727 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:48.728+0000 7f766b59e640 1 librados: init done 2024-03-29T11:26:48.727 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:48.728+0000 7f765affd640 1 -- 172.21.15.84:0/2737032010 <== mon.2 v1:172.21.15.84:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f765c001b10 con 0x7f7664036dc0 2024-03-29T11:26:48.727 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:48.728+0000 7f765affd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:26:48.727 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:48.728+0000 7f765affd640 1 -- 172.21.15.84:0/2737032010 <== mon.2 v1:172.21.15.84:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4236868694 0 0) 0x7f765c0038c0 con 0x7f7664036dc0 2024-03-29T11:26:48.727 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:48.728+0000 7f765affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:48.727 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:48.728+0000 7f765affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:26:48.727 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:48.728+0000 7f765affd640 10 monclient: dump: 2024-03-29T11:26:48.727 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T11:26:48.727 INFO:teuthology.misc.health.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:26:48.727 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T11:18:53.864025+0000 2024-03-29T11:26:48.728 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T11:18:53.864025+0000 2024-03-29T11:26:48.728 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:48.728 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:26:48.728 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T11:26:48.728 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T11:26:48.728 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T11:26:48.728 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:26:48.728 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:48.729+0000 7f7668d99640 4 set_mon_vals no callback set 2024-03-29T11:26:48.728 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:48.731+0000 7f765affd640 1 -- 172.21.15.84:0/2737032010 <== mon.2 v1:172.21.15.84:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4033942909 0 0) 0x7f765c003a20 con 0x7f7664036dc0 2024-03-29T11:26:48.728 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:48.731+0000 7f765affd640 1 -- 172.21.15.84:0/2737032010 <== mon.2 v1:172.21.15.84:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2808139587 0 0) 0x7f765c084a20 con 0x7f7664036dc0 2024-03-29T11:26:48.728 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:48.731+0000 7f766a59c640 10 monclient: get_auth_request con 0x7f764c06e150 auth_method 0 2024-03-29T11:26:48.728 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:48.732+0000 7f765affd640 10 client.4520.objecter ms_dispatch 0x7f766c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:26:48.728 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:48.732+0000 7f765affd640 3 client.4520.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:26:48.728 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:48.732+0000 7f765affd640 3 client.4520.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:26:48.729 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:48.732+0000 7f765affd640 20 client.4520.objecter dump_active .. 0 homeless 2024-03-29T11:26:48.729 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:48.732+0000 7f765affd640 10 client.4520.objecter ms_handle_connect 0x7f764c06e150 2024-03-29T11:26:48.729 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:48.733+0000 7f766b59e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:26:48.729 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:48.733+0000 7f766b59e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:26:48.729 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:48.733+0000 7f766b59e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-29T11:26:48.729 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:48.733+0000 7f766b59e640 1 -- 172.21.15.84:0/2737032010 --> v1:172.21.15.84:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7650005080 con 0x7f7664036dc0 2024-03-29T11:26:48.734 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:48.735+0000 7f7669d9b640 1 -- 172.21.15.84:0/2737032010 >> v1:172.21.15.84:6790/0 conn(0x7f7664036dc0 legacy=0x7f766402ec10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-03-29T11:26:48.734 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:48.735+0000 7f7669d9b640 1 -- 172.21.15.84:0/2737032010 >> v1:172.21.15.84:6790/0 conn(0x7f7664036dc0 legacy=0x7f766402ec10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-29T11:26:48.734 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:48.735+0000 7f7669d9b640 1 --1- 172.21.15.84:0/2737032010 >> v1:172.21.15.84:6790/0 conn(0x7f7664036dc0 0x7f766402ec10 :-1 s=OPENED pgs=721507 cs=1 l=1).handle_message read tag failed 2024-03-29T11:26:48.734 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:48.735+0000 7f7669d9b640 1 --1- 172.21.15.84:0/2737032010 >> v1:172.21.15.84:6790/0 conn(0x7f7664036dc0 0x7f766402ec10 :-1 s=OPENED pgs=721507 cs=1 l=1).fault on lossy channel, failing 2024-03-29T11:26:48.734 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:48.735+0000 7f765affd640 10 monclient: ms_handle_reset current mon v1:172.21.15.84:6790/0 2024-03-29T11:26:48.734 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:48.735+0000 7f765affd640 10 monclient: _reopen_session rank -1 2024-03-29T11:26:48.734 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:48.735+0000 7f765affd640 1 -- 172.21.15.84:0/2737032010 >> v1:172.21.15.84:6790/0 conn(0x7f7664036dc0 legacy=0x7f766402ec10 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-29T11:26:48.734 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:48.735+0000 7f765affd640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:26:48.734 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:48.736+0000 7f765affd640 10 monclient(hunting): picked mon.c con 0x7f764c072eb0 addr v1:172.21.15.84:6790/0 2024-03-29T11:26:48.734 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:48.736+0000 7f765affd640 10 monclient(hunting): picked mon.a con 0x7f76640cf4d0 addr v1:172.21.15.84:6789/0 2024-03-29T11:26:48.735 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:48.737+0000 7f765affd640 10 monclient(hunting): picked mon.b con 0x7f7664031eb0 addr v1:172.21.15.179:6789/0 2024-03-29T11:26:48.735 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:48.737+0000 7f765affd640 1 -- 172.21.15.84:0/2737032010 --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f76640ca7d0 con 0x7f76640cf4d0 2024-03-29T11:26:48.735 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:48.740+0000 7f765affd640 1 -- 172.21.15.84:0/2737032010 --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f764c004eb0 con 0x7f7664031eb0 2024-03-29T11:26:48.735 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:48.740+0000 7f765affd640 1 -- 172.21.15.84:0/2737032010 --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f76640cc970 con 0x7f764c072eb0 2024-03-29T11:26:48.735 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:48.740+0000 7f765affd640 10 monclient(hunting): _renew_subs 2024-03-29T11:26:48.735 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:48.741+0000 7f765affd640 10 client.4520.objecter ms_handle_connect 0x7f764c072eb0 2024-03-29T11:26:48.735 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:48.741+0000 7f765affd640 10 client.4520.objecter resend_mon_ops 2024-03-29T11:26:48.735 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:48.742+0000 7f765affd640 10 client.4520.objecter ms_handle_connect 0x7f76640cf4d0 2024-03-29T11:26:48.735 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:48.742+0000 7f765affd640 10 client.4520.objecter resend_mon_ops 2024-03-29T11:26:48.735 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:48.742+0000 7f765affd640 1 -- 172.21.15.84:0/2737032010 <== mon.0 v1:172.21.15.84:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1867690101 0 0) 0x7f76640ca7d0 con 0x7f76640cf4d0 2024-03-29T11:26:48.735 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:48.742+0000 7f765affd640 10 monclient(hunting): _negotiate 2024-03-29T11:26:48.735 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:48.743+0000 7f765affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:26:48.735 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:48.743+0000 7f765affd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-03-29T11:26:48.736 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:48.743+0000 7f765affd640 1 -- 172.21.15.84:0/2737032010 --> v1:172.21.15.84:6789/0 -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f764c07e4c0 con 0x7f76640cf4d0 2024-03-29T11:26:48.736 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:48.743+0000 7f765affd640 1 -- 172.21.15.84:0/2737032010 <== mon.0 v1:172.21.15.84:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 233+0+0 (unknown 1887031868 0 0) 0x7f764c07e4c0 con 0x7f76640cf4d0 2024-03-29T11:26:48.736 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:48.743+0000 7f765affd640 1 -- 172.21.15.84:0/2737032010 >> v1:172.21.15.84:6790/0 conn(0x7f764c072eb0 legacy=0x7f764c0752a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:48.736 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:48.743+0000 7f765affd640 1 -- 172.21.15.84:0/2737032010 >> v1:172.21.15.179:6789/0 conn(0x7f7664031eb0 legacy=0x7f764c07c8d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:26:48.736 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:48.744+0000 7f765affd640 10 monclient: _finish_hunting 0 2024-03-29T11:26:48.736 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:48.744+0000 7f765affd640 1 monclient: found mon.a 2024-03-29T11:26:48.736 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:48.744+0000 7f765affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:26:48.736 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:48.744+0000 7f765affd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.84:6789/0 2024-03-29T11:26:48.736 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:48.744+0000 7f765affd640 1 -- 172.21.15.84:0/2737032010 --> v1:172.21.15.84:6789/0 -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f764c081550 con 0x7f76640cf4d0 2024-03-29T11:26:48.736 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:48.744+0000 7f765affd640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:26:48.736 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:48.744+0000 7f765affd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.84:6789/0 2024-03-29T11:26:48.736 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:48.744+0000 7f765affd640 1 -- 172.21.15.84:0/2737032010 --> v1:172.21.15.84:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f764c004eb0 con 0x7f76640cf4d0 2024-03-29T11:26:48.736 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:48.744+0000 7f765affd640 10 monclient: _finish_auth 0 2024-03-29T11:26:48.736 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:48.744+0000 7f765affd640 10 monclient: _check_auth_tickets 2024-03-29T11:26:48.737 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:48.744+0000 7f765affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:26:48.737 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:48.744+0000 7f765affd640 1 -- 172.21.15.84:0/2737032010 <== mon.0 v1:172.21.15.84:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4236868694 0 0) 0x7f7660001c90 con 0x7f76640cf4d0 2024-03-29T11:26:48.737 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:48.744+0000 7f765affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:48.737 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:48.744+0000 7f765affd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:26:48.737 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:48.745+0000 7f765affd640 10 monclient: dump: 2024-03-29T11:26:48.737 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T11:26:48.737 INFO:teuthology.misc.health.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:26:48.737 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T11:18:53.864025+0000 2024-03-29T11:26:48.737 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T11:18:53.864025+0000 2024-03-29T11:26:48.737 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:48.737 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:26:48.737 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T11:26:48.737 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T11:26:48.738 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T11:26:48.738 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:26:48.738 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:48.746+0000 7f765affd640 1 -- 172.21.15.84:0/2737032010 <== mon.0 v1:172.21.15.84:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f76600038a0 con 0x7f76640cf4d0 2024-03-29T11:26:48.738 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:48.746+0000 7f765affd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:26:48.738 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:48.746+0000 7f765affd640 1 -- 172.21.15.84:0/2737032010 <== mon.0 v1:172.21.15.84:6789/0 5 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4033942909 0 0) 0x7f7660019880 con 0x7f76640cf4d0 2024-03-29T11:26:48.738 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:48.747+0000 7f765bfff640 4 set_mon_vals no callback set 2024-03-29T11:26:48.741 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:48.754+0000 7f765affd640 1 -- 172.21.15.84:0/2737032010 <== mon.0 v1:172.21.15.84:6789/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f7660001080 con 0x7f76640cf4d0 2024-03-29T11:26:48.741 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:48.754+0000 7f765affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:26:48.741 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:48.754+0000 7f765affd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:26:49.098 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:49.110+0000 7f766b59e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:26:49.099 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:49.110+0000 7f766b59e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:26:49.099 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:49.110+0000 7f766b59e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.84:6789/0 2024-03-29T11:26:49.099 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:49.110+0000 7f766b59e640 1 -- 172.21.15.84:0/2737032010 --> v1:172.21.15.84:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7650005330 con 0x7f76640cf4d0 2024-03-29T11:26:49.103 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:49.115+0000 7f765affd640 1 -- 172.21.15.84:0/2737032010 <== mon.0 v1:172.21.15.84:6789/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4209477549) 0x7f7660052830 con 0x7f76640cf4d0 2024-03-29T11:26:49.103 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:49.115+0000 7f765affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:26:49.103 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:49.115+0000 7f765affd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:26:49.104 INFO:teuthology.misc.health.smithi084.stdout: 2024-03-29T11:26:49.104 INFO:teuthology.misc.health.smithi084.stdout:{"epoch":10,"fsid":"2c3e002f-9eb0-4860-853b-4ef1ef562213","created":"2024-03-29T11:19:32.029169+0000","modified":"2024-03-29T11:19:34.200299+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:34.200299+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"5c39e276-2034-4716-bbc0-57a634eefa92","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"cd580e99-109f-4bf1-88c0-a5987bc8b9ed","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"11e60a5a-051d-453f-a190-66387cb35592","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"2c329fbe-bbc4-4030-bf87-d711cad04679","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f4213d4a-bd36-4433-91ae-d7e2b02481d3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"f0cf44ea-6055-48af-805b-aed858102ff9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"d895eaa0-cc0b-4ee4-9b5a-2dfcd89a60b7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"9bc5a7bc-8f52-4db9-96d6-de5e68d52eaa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:26:49.108 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:49.119+0000 7f766b59e640 10 librados: watch_flush enter 2024-03-29T11:26:49.108 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:49.119+0000 7f766b59e640 10 librados: watch_flush exit 2024-03-29T11:26:49.108 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:49.119+0000 7f766b59e640 20 client.4520.objecter shutdown clearing up homeless session... 2024-03-29T11:26:49.108 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:49.119+0000 7f766b59e640 10 client.4520.objecter successfully canceled tick 2024-03-29T11:26:49.108 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:49.120+0000 7f766b59e640 1 -- 172.21.15.84:0/2737032010 >> v1:172.21.15.179:6816/2981134980 conn(0x7f764c06e150 legacy=0x7f764c070610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:49.109 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:49.120+0000 7f766b59e640 10 monclient: shutdown 2024-03-29T11:26:49.109 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:49.120+0000 7f766b59e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:26:49.109 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:49.120+0000 7f766b59e640 1 -- 172.21.15.84:0/2737032010 >> v1:172.21.15.84:6789/0 conn(0x7f76640cf4d0 legacy=0x7f764c078cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:49.109 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:49.120+0000 7f766b59e640 1 -- 172.21.15.84:0/2737032010 shutdown_connections 2024-03-29T11:26:49.109 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:49.120+0000 7f766b59e640 1 -- 172.21.15.84:0/2737032010 >> 172.21.15.84:0/2737032010 conn(0x7f766403d480 msgr2=0x7f766403d860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:26:49.109 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:49.120+0000 7f766b59e640 1 -- 172.21.15.84:0/2737032010 shutdown_connections 2024-03-29T11:26:49.109 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:49.121+0000 7f766b59e640 1 -- 172.21.15.84:0/2737032010 wait complete. 2024-03-29T11:26:49.109 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:49.121+0000 7f766b59e640 1 librados: shutdown 2024-03-29T11:26:49.127 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:26:55.933 DEBUG:teuthology.orchestra.run.smithi084:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:26:56.085 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:56.098+0000 7f2345eeb640 10 monclient: get_monmap_and_config 2024-03-29T11:26:56.085 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:56.098+0000 7f2345eeb640 10 monclient: build_initial_monmap 2024-03-29T11:26:56.085 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:56.098+0000 7f2345eeb640 10 monclient: monmap: 2024-03-29T11:26:56.085 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-29T11:26:56.086 INFO:teuthology.misc.health.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:26:56.086 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T11:26:56.099154+0000 2024-03-29T11:26:56.086 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T11:26:56.099154+0000 2024-03-29T11:26:56.086 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-29T11:26:56.086 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:26:56.086 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-29T11:26:56.086 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.noname-c 2024-03-29T11:26:56.086 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-b 2024-03-29T11:26:56.086 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:26:56.087 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:56.099+0000 7f2345eeb640 1 Processor -- start 2024-03-29T11:26:56.087 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:56.099+0000 7f2345eeb640 1 -- start start 2024-03-29T11:26:56.087 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:56.099+0000 7f2345eeb640 10 monclient: init 2024-03-29T11:26:56.087 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:56.100+0000 7f2345eeb640 10 monclient: _reopen_session rank -1 2024-03-29T11:26:56.087 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:56.100+0000 7f2345eeb640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:26:56.088 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:56.101+0000 7f2345eeb640 10 monclient(hunting): picked mon.noname-c con 0x7f23380389e0 addr v1:172.21.15.179:6789/0 2024-03-29T11:26:56.088 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:56.101+0000 7f2345eeb640 10 monclient(hunting): picked mon.noname-b con 0x7f2338038dc0 addr v1:172.21.15.84:6790/0 2024-03-29T11:26:56.088 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:56.101+0000 7f2345eeb640 10 monclient(hunting): picked mon.noname-a con 0x7f2338030660 addr v1:172.21.15.84:6789/0 2024-03-29T11:26:56.088 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:56.101+0000 7f2345eeb640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f23380c54c0 con 0x7f2338030660 2024-03-29T11:26:56.088 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:56.101+0000 7f2345eeb640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2338036430 con 0x7f23380389e0 2024-03-29T11:26:56.089 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:56.101+0000 7f2345eeb640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2338039840 con 0x7f2338038dc0 2024-03-29T11:26:56.089 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:56.101+0000 7f2345eeb640 10 monclient(hunting): _renew_subs 2024-03-29T11:26:56.089 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:56.101+0000 7f2345eeb640 10 monclient(hunting): authenticate will time out at 1277.395732s 2024-03-29T11:26:56.089 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:56.102+0000 7f23456ea640 1 --1- >> v1:172.21.15.84:6789/0 conn(0x7f2338030660 0x7f23380d3170 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.84:6789/0 says I am v1:172.21.15.84:43440/0 (socket says 172.21.15.84:43440) 2024-03-29T11:26:56.089 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:56.102+0000 7f23456ea640 1 -- 172.21.15.84:0/1683657907 learned_addr learned my addr 172.21.15.84:0/1683657907 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-29T11:26:56.089 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:56.102+0000 7f233f7fe640 1 -- 172.21.15.84:0/1683657907 <== mon.0 v1:172.21.15.84:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1500698985 0 0) 0x7f23380c54c0 con 0x7f2338030660 2024-03-29T11:26:56.090 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:56.103+0000 7f233f7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:26:56.090 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:56.103+0000 7f233f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:26:56.090 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:56.103+0000 7f233f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:26:56.090 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:56.103+0000 7f233f7fe640 10 monclient(hunting): my global_id is 4500 2024-03-29T11:26:56.090 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:56.103+0000 7f233f7fe640 1 -- 172.21.15.84:0/1683657907 --> v1:172.21.15.84:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2330003520 con 0x7f2338030660 2024-03-29T11:26:56.091 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:56.103+0000 7f233f7fe640 1 -- 172.21.15.84:0/1683657907 <== mon.0 v1:172.21.15.84:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1330207379 0 0) 0x7f2330003520 con 0x7f2338030660 2024-03-29T11:26:56.091 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:56.103+0000 7f233f7fe640 1 -- 172.21.15.84:0/1683657907 >> v1:172.21.15.84:6790/0 conn(0x7f2338038dc0 legacy=0x7f23380cadd0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:26:56.091 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:56.104+0000 7f233f7fe640 1 -- 172.21.15.84:0/1683657907 >> v1:172.21.15.179:6789/0 conn(0x7f23380389e0 legacy=0x7f2338033520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:56.092 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:56.104+0000 7f233f7fe640 10 monclient: _finish_hunting 0 2024-03-29T11:26:56.092 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:56.104+0000 7f233f7fe640 1 monclient: found mon.noname-a 2024-03-29T11:26:56.092 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:56.104+0000 7f233f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:26:56.092 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:56.104+0000 7f233f7fe640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.84:6789/0 2024-03-29T11:26:56.092 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:56.104+0000 7f233f7fe640 1 -- 172.21.15.84:0/1683657907 --> v1:172.21.15.84:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2338035690 con 0x7f2338030660 2024-03-29T11:26:56.092 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:56.104+0000 7f233f7fe640 10 monclient: _finish_auth 0 2024-03-29T11:26:56.092 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:56.104+0000 7f233f7fe640 10 monclient: _check_auth_tickets 2024-03-29T11:26:56.092 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:56.104+0000 7f233f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:26:56.092 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:56.104+0000 7f233f7fe640 1 -- 172.21.15.84:0/1683657907 <== mon.0 v1:172.21.15.84:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4236868694 0 0) 0x7f2334001690 con 0x7f2338030660 2024-03-29T11:26:56.092 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:56.104+0000 7f233f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:56.092 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:56.104+0000 7f233f7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:26:56.092 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:56.104+0000 7f233f7fe640 10 monclient: dump: 2024-03-29T11:26:56.092 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T11:26:56.092 INFO:teuthology.misc.health.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:26:56.093 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T11:18:53.864025+0000 2024-03-29T11:26:56.093 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T11:18:53.864025+0000 2024-03-29T11:26:56.093 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:56.093 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:26:56.093 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T11:26:56.093 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T11:26:56.093 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T11:26:56.093 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:26:56.094 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:56.105+0000 7f233f7fe640 1 -- 172.21.15.84:0/1683657907 <== mon.0 v1:172.21.15.84:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f23340019a0 con 0x7f2338030660 2024-03-29T11:26:56.094 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:56.105+0000 7f233f7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:26:56.094 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:56.105+0000 7f233f7fe640 1 -- 172.21.15.84:0/1683657907 <== mon.0 v1:172.21.15.84:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4236868694 0 0) 0x7f2334001d30 con 0x7f2338030660 2024-03-29T11:26:56.094 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:56.105+0000 7f233f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:56.094 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:56.105+0000 7f233f7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:26:56.095 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:56.105+0000 7f233f7fe640 10 monclient: dump: 2024-03-29T11:26:56.095 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T11:26:56.095 INFO:teuthology.misc.health.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:26:56.095 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T11:18:53.864025+0000 2024-03-29T11:26:56.095 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T11:18:53.864025+0000 2024-03-29T11:26:56.095 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:56.095 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:26:56.095 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T11:26:56.095 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T11:26:56.095 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T11:26:56.095 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:26:56.095 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:56.107+0000 7f2345eeb640 5 monclient: authenticate success, global_id 4500 2024-03-29T11:26:56.095 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:56.107+0000 7f2345eeb640 10 monclient: get_monmap_and_config success 2024-03-29T11:26:56.095 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:56.107+0000 7f2345eeb640 4 set_mon_vals no callback set 2024-03-29T11:26:56.097 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:56.108+0000 7f2345eeb640 10 monclient: shutdown 2024-03-29T11:26:56.097 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:56.108+0000 7f2345eeb640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:26:56.097 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:56.108+0000 7f2345eeb640 1 -- 172.21.15.84:0/1683657907 >> v1:172.21.15.84:6789/0 conn(0x7f2338030660 legacy=0x7f23380d3170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:56.097 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:56.108+0000 7f2345eeb640 1 -- 172.21.15.84:0/1683657907 shutdown_connections 2024-03-29T11:26:56.097 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:56.108+0000 7f2345eeb640 1 -- 172.21.15.84:0/1683657907 >> 172.21.15.84:0/1683657907 conn(0x7f2338005c00 msgr2=0x7f2338006010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:26:56.097 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:56.108+0000 7f2345eeb640 1 -- 172.21.15.84:0/1683657907 shutdown_connections 2024-03-29T11:26:56.097 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:56.109+0000 7f2345eeb640 1 -- 172.21.15.84:0/1683657907 wait complete. 2024-03-29T11:26:56.097 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:56.109+0000 7f2345eeb640 10 monclient: build_initial_monmap 2024-03-29T11:26:56.097 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:56.110+0000 7f2345eeb640 10 monclient: monmap: 2024-03-29T11:26:56.097 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-29T11:26:56.097 INFO:teuthology.misc.health.smithi084.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:26:56.097 INFO:teuthology.misc.health.smithi084.stderr:last_changed 0.000000 2024-03-29T11:26:56.097 INFO:teuthology.misc.health.smithi084.stderr:created 0.000000 2024-03-29T11:26:56.098 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-29T11:26:56.098 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:26:56.098 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-29T11:26:56.098 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.noname-b 2024-03-29T11:26:56.098 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-c 2024-03-29T11:26:56.098 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:26:56.099 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:56.110+0000 7f2345eeb640 1 librados: starting msgr at 2024-03-29T11:26:56.099 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:56.110+0000 7f2345eeb640 1 librados: starting objecter 2024-03-29T11:26:56.099 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:56.110+0000 7f2345eeb640 1 Processor -- start 2024-03-29T11:26:56.099 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:56.111+0000 7f2345eeb640 1 -- start start 2024-03-29T11:26:56.099 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:56.111+0000 7f2345eeb640 1 librados: setting wanted keys 2024-03-29T11:26:56.100 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:56.111+0000 7f2345eeb640 1 librados: calling monclient init 2024-03-29T11:26:56.100 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:56.111+0000 7f2345eeb640 10 monclient: init 2024-03-29T11:26:56.100 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:56.112+0000 7f2345eeb640 10 monclient: _reopen_session rank -1 2024-03-29T11:26:56.100 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:56.112+0000 7f2345eeb640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:26:56.100 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:56.112+0000 7f2345eeb640 10 monclient(hunting): picked mon.noname-a con 0x7f2338030660 addr v1:172.21.15.84:6789/0 2024-03-29T11:26:56.100 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:56.112+0000 7f2344ee9640 1 --1- >> v1:172.21.15.84:6789/0 conn(0x7f2338030660 0x7f23381604e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.84:6789/0 says I am v1:172.21.15.84:44080/0 (socket says 172.21.15.84:44080) 2024-03-29T11:26:56.103 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:56.115+0000 7f2345eeb640 10 monclient(hunting): picked mon.noname-c con 0x7f23380389e0 addr v1:172.21.15.84:6790/0 2024-03-29T11:26:56.104 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:56.115+0000 7f2345eeb640 10 monclient(hunting): picked mon.noname-b con 0x7f2338160bf0 addr v1:172.21.15.179:6789/0 2024-03-29T11:26:56.104 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:56.115+0000 7f2345eeb640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f233815ffe0 con 0x7f2338030660 2024-03-29T11:26:56.104 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:56.115+0000 7f2345eeb640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f23380c71e0 con 0x7f2338160bf0 2024-03-29T11:26:56.104 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:56.115+0000 7f2345eeb640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f23380c79c0 con 0x7f23380389e0 2024-03-29T11:26:56.104 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:56.115+0000 7f2345eeb640 10 monclient(hunting): _renew_subs 2024-03-29T11:26:56.104 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:56.115+0000 7f2345eeb640 10 monclient(hunting): authenticate will time out at 1277.409453s 2024-03-29T11:26:56.104 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:56.115+0000 7f2344ee9640 1 -- 172.21.15.84:0/2041978762 learned_addr learned my addr 172.21.15.84:0/2041978762 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-29T11:26:56.104 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:56.115+0000 7f233d7fa640 10 client.?.objecter ms_handle_connect 0x7f2338030660 2024-03-29T11:26:56.104 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:56.115+0000 7f233d7fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:26:56.104 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:56.115+0000 7f233d7fa640 1 -- 172.21.15.84:0/2041978762 <== mon.0 v1:172.21.15.84:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1383820635 0 0) 0x7f233815ffe0 con 0x7f2338030660 2024-03-29T11:26:56.104 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:56.115+0000 7f233d7fa640 10 monclient(hunting): _negotiate 2024-03-29T11:26:56.104 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:56.115+0000 7f233d7fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:26:56.105 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:56.115+0000 7f233d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:26:56.105 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:56.115+0000 7f233d7fa640 10 monclient(hunting): my global_id is 4503 2024-03-29T11:26:56.105 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:56.115+0000 7f233d7fa640 1 -- 172.21.15.84:0/2041978762 --> v1:172.21.15.84:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2324004eb0 con 0x7f2338030660 2024-03-29T11:26:56.105 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:56.116+0000 7f233d7fa640 1 -- 172.21.15.84:0/2041978762 <== mon.0 v1:172.21.15.84:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2371483269 0 0) 0x7f2324004eb0 con 0x7f2338030660 2024-03-29T11:26:56.105 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:56.116+0000 7f233d7fa640 1 -- 172.21.15.84:0/2041978762 --> v1:172.21.15.84:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f233815ffe0 con 0x7f2338030660 2024-03-29T11:26:56.105 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:56.116+0000 7f233d7fa640 1 -- 172.21.15.84:0/2041978762 <== mon.0 v1:172.21.15.84:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4236868694 0 0) 0x7f234004fdd0 con 0x7f2338030660 2024-03-29T11:26:56.105 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:56.116+0000 7f233d7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:56.105 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:56.116+0000 7f233d7fa640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:26:56.105 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:56.116+0000 7f233d7fa640 10 monclient(hunting): dump: 2024-03-29T11:26:56.105 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T11:26:56.105 INFO:teuthology.misc.health.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:26:56.105 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T11:18:53.864025+0000 2024-03-29T11:26:56.105 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T11:18:53.864025+0000 2024-03-29T11:26:56.106 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:56.106 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:26:56.106 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T11:26:56.106 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T11:26:56.106 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T11:26:56.106 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:26:56.106 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:56.116+0000 7f233d7fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:26:56.106 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:56.116+0000 7f233d7fa640 10 client.?.objecter ms_handle_connect 0x7f23380389e0 2024-03-29T11:26:56.106 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:56.116+0000 7f233d7fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:26:56.106 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:56.116+0000 7f233d7fa640 1 -- 172.21.15.84:0/2041978762 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 284293415 0 0) 0x7f23380c79c0 con 0x7f23380389e0 2024-03-29T11:26:56.107 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:56.116+0000 7f233d7fa640 10 monclient(hunting): _negotiate 2024-03-29T11:26:56.107 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:56.116+0000 7f233d7fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:26:56.107 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:56.116+0000 7f233d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:26:56.107 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:56.116+0000 7f233d7fa640 10 monclient(hunting): my global_id is 4523 2024-03-29T11:26:56.107 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:56.116+0000 7f233d7fa640 1 -- 172.21.15.84:0/2041978762 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2324004eb0 con 0x7f23380389e0 2024-03-29T11:26:56.107 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:56.116+0000 7f233d7fa640 1 -- 172.21.15.84:0/2041978762 <== mon.0 v1:172.21.15.84:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 293606424 0 0) 0x7f233815ffe0 con 0x7f2338030660 2024-03-29T11:26:56.107 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:56.116+0000 7f233d7fa640 1 -- 172.21.15.84:0/2041978762 >> v1:172.21.15.84:6790/0 conn(0x7f23380389e0 legacy=0x7f23380c7f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:56.107 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:56.116+0000 7f233d7fa640 1 -- 172.21.15.84:0/2041978762 >> v1:172.21.15.179:6789/0 conn(0x7f2338160bf0 legacy=0x7f23381682e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:26:56.112 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:56.118+0000 7f233d7fa640 10 monclient: _finish_hunting 0 2024-03-29T11:26:56.112 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:56.118+0000 7f233d7fa640 1 monclient: found mon.a 2024-03-29T11:26:56.112 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:56.118+0000 7f233d7fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:26:56.112 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:56.118+0000 7f233d7fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.84:6789/0 2024-03-29T11:26:56.113 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:56.118+0000 7f233d7fa640 1 -- 172.21.15.84:0/2041978762 --> v1:172.21.15.84:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23380c7b50 con 0x7f2338030660 2024-03-29T11:26:56.113 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:56.118+0000 7f233d7fa640 10 monclient: _finish_auth 0 2024-03-29T11:26:56.113 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:56.118+0000 7f233d7fa640 10 monclient: _check_auth_tickets 2024-03-29T11:26:56.113 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:56.118+0000 7f233d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:26:56.113 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:56.118+0000 7f2345eeb640 5 monclient: authenticate success, global_id 4503 2024-03-29T11:26:56.113 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:56.118+0000 7f2345eeb640 10 monclient: _renew_subs 2024-03-29T11:26:56.113 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:56.118+0000 7f2345eeb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.84:6789/0 2024-03-29T11:26:56.113 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:56.118+0000 7f2345eeb640 1 -- 172.21.15.84:0/2041978762 --> v1:172.21.15.84:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f23380c5650 con 0x7f2338030660 2024-03-29T11:26:56.113 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:56.118+0000 7f2345eeb640 10 client.4503.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:26:56.113 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:56.118+0000 7f2345eeb640 10 monclient: _renew_subs 2024-03-29T11:26:56.113 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:56.118+0000 7f2345eeb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.84:6789/0 2024-03-29T11:26:56.113 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:56.118+0000 7f2345eeb640 1 -- 172.21.15.84:0/2041978762 --> v1:172.21.15.84:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f23380c5bc0 con 0x7f2338030660 2024-03-29T11:26:56.113 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:56.118+0000 7f2345eeb640 1 librados: init done 2024-03-29T11:26:56.114 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:56.120+0000 7f233d7fa640 1 -- 172.21.15.84:0/2041978762 <== mon.0 v1:172.21.15.84:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f234004f400 con 0x7f2338030660 2024-03-29T11:26:56.114 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:56.120+0000 7f233d7fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:26:56.114 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:56.120+0000 7f233d7fa640 1 -- 172.21.15.84:0/2041978762 <== mon.0 v1:172.21.15.84:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4236868694 0 0) 0x7f2340075710 con 0x7f2338030660 2024-03-29T11:26:56.114 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:56.120+0000 7f233d7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:56.114 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:56.120+0000 7f233d7fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:26:56.114 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:56.120+0000 7f232affd640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:26:56.114 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:56.120+0000 7f233d7fa640 10 monclient: dump: 2024-03-29T11:26:56.114 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T11:26:56.114 INFO:teuthology.misc.health.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:26:56.114 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T11:18:53.864025+0000 2024-03-29T11:26:56.114 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T11:18:53.864025+0000 2024-03-29T11:26:56.114 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:56.114 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:26:56.115 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T11:26:56.115 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T11:26:56.115 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T11:26:56.115 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:26:56.115 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:56.120+0000 7f233effd640 4 set_mon_vals no callback set 2024-03-29T11:26:56.115 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:56.121+0000 7f232affd640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:26:56.115 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:56.121+0000 7f232affd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.84:6789/0 2024-03-29T11:26:56.115 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:56.121+0000 7f232affd640 1 -- 172.21.15.84:0/2041978762 --> v1:172.21.15.84:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2320005080 con 0x7f2338030660 2024-03-29T11:26:56.115 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:56.121+0000 7f233d7fa640 1 -- 172.21.15.84:0/2041978762 <== mon.0 v1:172.21.15.84:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4033942909 0 0) 0x7f2340067bf0 con 0x7f2338030660 2024-03-29T11:26:56.115 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:56.122+0000 7f233d7fa640 1 -- 172.21.15.84:0/2041978762 <== mon.0 v1:172.21.15.84:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2808139587 0 0) 0x7f23400ff5a0 con 0x7f2338030660 2024-03-29T11:26:56.115 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:56.122+0000 7f233d7fa640 10 client.4503.objecter ms_dispatch 0x7f2340000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:26:56.116 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:56.122+0000 7f233d7fa640 3 client.4503.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:26:56.116 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:56.122+0000 7f233d7fa640 3 client.4503.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:26:56.116 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:56.122+0000 7f233d7fa640 20 client.4503.objecter dump_active .. 0 homeless 2024-03-29T11:26:56.116 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:56.122+0000 7f233ffff640 10 monclient: get_auth_request con 0x7f232406db80 auth_method 0 2024-03-29T11:26:56.116 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:56.124+0000 7f233d7fa640 10 client.4503.objecter ms_handle_connect 0x7f232406db80 2024-03-29T11:26:56.122 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:56.134+0000 7f233d7fa640 1 -- 172.21.15.84:0/2041978762 <== mon.0 v1:172.21.15.84:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f23400cae80 con 0x7f2338030660 2024-03-29T11:26:56.122 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:56.134+0000 7f233d7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:26:56.122 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:56.134+0000 7f233d7fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:26:56.442 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:56.454+0000 7f232affd640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:26:56.442 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:56.454+0000 7f232affd640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:26:56.442 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:56.455+0000 7f232affd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.84:6789/0 2024-03-29T11:26:56.442 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:56.455+0000 7f232affd640 1 -- 172.21.15.84:0/2041978762 --> v1:172.21.15.84:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2320005330 con 0x7f2338030660 2024-03-29T11:26:56.443 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:56.457+0000 7f233d7fa640 1 -- 172.21.15.84:0/2041978762 <== mon.0 v1:172.21.15.84:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4209477549) 0x7f23400ccdb0 con 0x7f2338030660 2024-03-29T11:26:56.444 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:56.457+0000 7f233d7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:26:56.444 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:56.457+0000 7f233d7fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:26:56.444 INFO:teuthology.misc.health.smithi084.stdout: 2024-03-29T11:26:56.444 INFO:teuthology.misc.health.smithi084.stdout:{"epoch":10,"fsid":"2c3e002f-9eb0-4860-853b-4ef1ef562213","created":"2024-03-29T11:19:32.029169+0000","modified":"2024-03-29T11:19:34.200299+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:34.200299+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"5c39e276-2034-4716-bbc0-57a634eefa92","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"cd580e99-109f-4bf1-88c0-a5987bc8b9ed","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"11e60a5a-051d-453f-a190-66387cb35592","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"2c329fbe-bbc4-4030-bf87-d711cad04679","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f4213d4a-bd36-4433-91ae-d7e2b02481d3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"f0cf44ea-6055-48af-805b-aed858102ff9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"d895eaa0-cc0b-4ee4-9b5a-2dfcd89a60b7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"9bc5a7bc-8f52-4db9-96d6-de5e68d52eaa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:26:56.447 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:56.460+0000 7f232affd640 10 librados: watch_flush enter 2024-03-29T11:26:56.447 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:56.460+0000 7f232affd640 10 librados: watch_flush exit 2024-03-29T11:26:56.448 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:56.460+0000 7f232affd640 20 client.4503.objecter shutdown clearing up homeless session... 2024-03-29T11:26:56.448 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:56.460+0000 7f232affd640 10 client.4503.objecter successfully canceled tick 2024-03-29T11:26:56.448 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:56.460+0000 7f232affd640 1 -- 172.21.15.84:0/2041978762 >> v1:172.21.15.179:6816/2981134980 conn(0x7f232406db80 legacy=0x7f2324070040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:56.448 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:56.460+0000 7f232affd640 10 monclient: shutdown 2024-03-29T11:26:56.448 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:56.460+0000 7f232affd640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:26:56.448 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:56.460+0000 7f232affd640 1 -- 172.21.15.84:0/2041978762 >> v1:172.21.15.84:6789/0 conn(0x7f2338030660 legacy=0x7f23381604e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:56.449 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:56.461+0000 7f232affd640 1 -- 172.21.15.84:0/2041978762 shutdown_connections 2024-03-29T11:26:56.449 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:56.461+0000 7f232affd640 1 -- 172.21.15.84:0/2041978762 >> 172.21.15.84:0/2041978762 conn(0x7f2338005c00 msgr2=0x7f23380c47e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:26:56.449 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:56.461+0000 7f232affd640 1 -- 172.21.15.84:0/2041978762 shutdown_connections 2024-03-29T11:26:56.449 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:56.461+0000 7f232affd640 1 -- 172.21.15.84:0/2041978762 wait complete. 2024-03-29T11:26:56.449 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:26:56.461+0000 7f232affd640 1 librados: shutdown 2024-03-29T11:26:56.465 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:27:03.272 DEBUG:teuthology.orchestra.run.smithi084:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:27:03.424 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:03.437+0000 7fe52c863640 10 monclient: get_monmap_and_config 2024-03-29T11:27:03.425 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:03.437+0000 7fe52c863640 10 monclient: build_initial_monmap 2024-03-29T11:27:03.425 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:03.437+0000 7fe52c863640 10 monclient: monmap: 2024-03-29T11:27:03.425 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-29T11:27:03.425 INFO:teuthology.misc.health.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:27:03.425 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T11:27:03.438133+0000 2024-03-29T11:27:03.425 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T11:27:03.438133+0000 2024-03-29T11:27:03.425 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-29T11:27:03.425 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:27:03.425 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-29T11:27:03.425 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.noname-c 2024-03-29T11:27:03.425 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-b 2024-03-29T11:27:03.425 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:27:03.426 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:03.438+0000 7fe52c863640 1 Processor -- start 2024-03-29T11:27:03.426 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:03.438+0000 7fe52c863640 1 -- start start 2024-03-29T11:27:03.426 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:03.438+0000 7fe52c863640 10 monclient: init 2024-03-29T11:27:03.427 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:03.440+0000 7fe52c863640 10 monclient: _reopen_session rank -1 2024-03-29T11:27:03.428 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:03.440+0000 7fe52c863640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:27:03.428 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:03.440+0000 7fe52c863640 10 monclient(hunting): picked mon.noname-b con 0x7fe5281422b0 addr v1:172.21.15.84:6790/0 2024-03-29T11:27:03.429 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:03.440+0000 7fe52c863640 10 monclient(hunting): picked mon.noname-a con 0x7fe528142690 addr v1:172.21.15.84:6789/0 2024-03-29T11:27:03.429 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:03.440+0000 7fe52c863640 10 monclient(hunting): picked mon.noname-c con 0x7fe5280b17d0 addr v1:172.21.15.179:6789/0 2024-03-29T11:27:03.429 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:03.440+0000 7fe52c863640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe528142c80 con 0x7fe528142690 2024-03-29T11:27:03.429 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:03.440+0000 7fe52c863640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe5280aae40 con 0x7fe5280b17d0 2024-03-29T11:27:03.429 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:03.440+0000 7fe52c863640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe52813fbd0 con 0x7fe5281422b0 2024-03-29T11:27:03.429 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:03.440+0000 7fe52c863640 10 monclient(hunting): _renew_subs 2024-03-29T11:27:03.429 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:03.440+0000 7fe52c863640 10 monclient(hunting): authenticate will time out at 1284.734717s 2024-03-29T11:27:03.429 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:03.440+0000 7fe52659c640 1 --1- >> v1:172.21.15.84:6790/0 conn(0x7fe5281422b0 0x7fe52813f4c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.84:6790/0 says I am v1:172.21.15.84:41588/0 (socket says 172.21.15.84:41588) 2024-03-29T11:27:03.429 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:03.440+0000 7fe52659c640 1 -- 172.21.15.84:0/686234552 learned_addr learned my addr 172.21.15.84:0/686234552 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-29T11:27:03.433 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:03.441+0000 7fe52559a640 1 -- 172.21.15.84:0/686234552 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2379728873 0 0) 0x7fe52813fbd0 con 0x7fe5281422b0 2024-03-29T11:27:03.433 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:03.441+0000 7fe52559a640 10 monclient(hunting): _negotiate 2024-03-29T11:27:03.433 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:03.441+0000 7fe52559a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:27:03.433 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:03.441+0000 7fe52559a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:27:03.433 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:03.441+0000 7fe52559a640 10 monclient(hunting): my global_id is 4526 2024-03-29T11:27:03.433 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:03.441+0000 7fe52559a640 1 -- 172.21.15.84:0/686234552 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe50c003520 con 0x7fe5281422b0 2024-03-29T11:27:03.433 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:03.441+0000 7fe52559a640 1 -- 172.21.15.84:0/686234552 <== mon.1 v1:172.21.15.179:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1573122089 0 0) 0x7fe5280aae40 con 0x7fe5280b17d0 2024-03-29T11:27:03.433 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:03.441+0000 7fe52559a640 10 monclient(hunting): _negotiate 2024-03-29T11:27:03.433 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:03.441+0000 7fe52559a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:27:03.433 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:03.441+0000 7fe52559a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:27:03.433 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:03.441+0000 7fe52559a640 10 monclient(hunting): my global_id is 4447 2024-03-29T11:27:03.433 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:03.441+0000 7fe52559a640 1 -- 172.21.15.84:0/686234552 --> v1:172.21.15.179:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe52813fbd0 con 0x7fe5280b17d0 2024-03-29T11:27:03.434 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:03.441+0000 7fe52559a640 1 -- 172.21.15.84:0/686234552 <== mon.0 v1:172.21.15.84:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3365789549 0 0) 0x7fe528142c80 con 0x7fe528142690 2024-03-29T11:27:03.434 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:03.441+0000 7fe52559a640 10 monclient(hunting): _negotiate 2024-03-29T11:27:03.434 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:03.441+0000 7fe52559a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:27:03.434 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:03.441+0000 7fe52559a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:27:03.434 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:03.441+0000 7fe52559a640 10 monclient(hunting): my global_id is 4506 2024-03-29T11:27:03.434 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:03.441+0000 7fe52559a640 1 -- 172.21.15.84:0/686234552 --> v1:172.21.15.84:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe5280aae40 con 0x7fe528142690 2024-03-29T11:27:03.434 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:03.442+0000 7fe52559a640 1 -- 172.21.15.84:0/686234552 <== mon.2 v1:172.21.15.84:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2513947270 0 0) 0x7fe50c003520 con 0x7fe5281422b0 2024-03-29T11:27:03.435 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:03.442+0000 7fe52559a640 1 -- 172.21.15.84:0/686234552 >> v1:172.21.15.179:6789/0 conn(0x7fe5280b17d0 legacy=0x7fe5280b2ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:03.435 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:03.442+0000 7fe52559a640 1 -- 172.21.15.84:0/686234552 >> v1:172.21.15.84:6789/0 conn(0x7fe528142690 legacy=0x7fe5280b2590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:03.435 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:03.442+0000 7fe52559a640 10 monclient: _finish_hunting 0 2024-03-29T11:27:03.435 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:03.442+0000 7fe52559a640 1 monclient: found mon.noname-b 2024-03-29T11:27:03.435 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:03.442+0000 7fe52559a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:27:03.435 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:03.442+0000 7fe52559a640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.84:6790/0 2024-03-29T11:27:03.435 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:03.442+0000 7fe52559a640 1 -- 172.21.15.84:0/686234552 --> v1:172.21.15.84:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe528151940 con 0x7fe5281422b0 2024-03-29T11:27:03.435 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:03.442+0000 7fe52559a640 10 monclient: _finish_auth 0 2024-03-29T11:27:03.435 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:03.442+0000 7fe52559a640 10 monclient: _check_auth_tickets 2024-03-29T11:27:03.435 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:03.442+0000 7fe52559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:27:03.435 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:03.442+0000 7fe52559a640 1 -- 172.21.15.84:0/686234552 <== mon.2 v1:172.21.15.84:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4236868694 0 0) 0x7fe518001690 con 0x7fe5281422b0 2024-03-29T11:27:03.435 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:03.442+0000 7fe52559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:27:03.435 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:03.442+0000 7fe52559a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:27:03.435 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:03.442+0000 7fe52559a640 10 monclient: dump: 2024-03-29T11:27:03.436 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T11:27:03.436 INFO:teuthology.misc.health.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:27:03.436 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T11:18:53.864025+0000 2024-03-29T11:27:03.436 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T11:18:53.864025+0000 2024-03-29T11:27:03.436 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T11:27:03.436 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:27:03.436 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T11:27:03.436 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T11:27:03.436 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T11:27:03.436 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:27:03.436 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:03.442+0000 7fe52c863640 5 monclient: authenticate success, global_id 4526 2024-03-29T11:27:03.436 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:03.442+0000 7fe52c863640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:27:03.436 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:03.442+0000 7fe52559a640 1 -- 172.21.15.84:0/686234552 <== mon.2 v1:172.21.15.84:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe5180017f0 con 0x7fe5281422b0 2024-03-29T11:27:03.436 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:03.442+0000 7fe52559a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:27:03.436 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:03.442+0000 7fe52c863640 10 monclient: get_monmap_and_config success 2024-03-29T11:27:03.436 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:03.442+0000 7fe52559a640 1 -- 172.21.15.84:0/686234552 <== mon.2 v1:172.21.15.84:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4236868694 0 0) 0x7fe518001b80 con 0x7fe5281422b0 2024-03-29T11:27:03.437 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:03.442+0000 7fe52559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:27:03.437 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:03.442+0000 7fe52c863640 4 set_mon_vals no callback set 2024-03-29T11:27:03.437 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:03.442+0000 7fe52559a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:27:03.437 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:03.442+0000 7fe52559a640 10 monclient: dump: 2024-03-29T11:27:03.437 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T11:27:03.437 INFO:teuthology.misc.health.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:27:03.437 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T11:18:53.864025+0000 2024-03-29T11:27:03.437 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T11:18:53.864025+0000 2024-03-29T11:27:03.437 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T11:27:03.437 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:27:03.437 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T11:27:03.437 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T11:27:03.437 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T11:27:03.437 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:27:03.437 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:03.443+0000 7fe52c863640 10 monclient: shutdown 2024-03-29T11:27:03.438 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:03.443+0000 7fe52c863640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:27:03.438 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:03.443+0000 7fe52c863640 1 -- 172.21.15.84:0/686234552 >> v1:172.21.15.84:6790/0 conn(0x7fe5281422b0 legacy=0x7fe52813f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:03.438 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:03.443+0000 7fe52c863640 1 -- 172.21.15.84:0/686234552 shutdown_connections 2024-03-29T11:27:03.438 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:03.443+0000 7fe52c863640 1 -- 172.21.15.84:0/686234552 >> 172.21.15.84:0/686234552 conn(0x7fe528066a00 msgr2=0x7fe528066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:27:03.438 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:03.443+0000 7fe52c863640 1 -- 172.21.15.84:0/686234552 shutdown_connections 2024-03-29T11:27:03.438 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:03.444+0000 7fe52c863640 1 -- 172.21.15.84:0/686234552 wait complete. 2024-03-29T11:27:03.438 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:03.445+0000 7fe52c863640 10 monclient: build_initial_monmap 2024-03-29T11:27:03.438 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:03.445+0000 7fe52c863640 10 monclient: monmap: 2024-03-29T11:27:03.438 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-29T11:27:03.438 INFO:teuthology.misc.health.smithi084.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:27:03.438 INFO:teuthology.misc.health.smithi084.stderr:last_changed 0.000000 2024-03-29T11:27:03.438 INFO:teuthology.misc.health.smithi084.stderr:created 0.000000 2024-03-29T11:27:03.438 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-29T11:27:03.438 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:27:03.438 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-29T11:27:03.439 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.noname-b 2024-03-29T11:27:03.439 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-c 2024-03-29T11:27:03.439 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:27:03.439 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:03.445+0000 7fe52c863640 1 librados: starting msgr at 2024-03-29T11:27:03.439 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:03.445+0000 7fe52c863640 1 librados: starting objecter 2024-03-29T11:27:03.439 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:03.445+0000 7fe52c863640 1 Processor -- start 2024-03-29T11:27:03.439 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:03.445+0000 7fe52c863640 1 -- start start 2024-03-29T11:27:03.439 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:03.445+0000 7fe52c863640 1 librados: setting wanted keys 2024-03-29T11:27:03.439 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:03.445+0000 7fe52c863640 1 librados: calling monclient init 2024-03-29T11:27:03.439 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:03.445+0000 7fe52c863640 10 monclient: init 2024-03-29T11:27:03.444 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:03.446+0000 7fe52c863640 10 monclient: _reopen_session rank -1 2024-03-29T11:27:03.444 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:03.446+0000 7fe52c863640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:27:03.444 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:03.446+0000 7fe52c863640 10 monclient(hunting): picked mon.noname-a con 0x7fe5280b17d0 addr v1:172.21.15.84:6789/0 2024-03-29T11:27:03.444 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:03.446+0000 7fe52c863640 10 monclient(hunting): picked mon.noname-b con 0x7fe5281422b0 addr v1:172.21.15.179:6789/0 2024-03-29T11:27:03.444 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:03.446+0000 7fe52659c640 1 --1- >> v1:172.21.15.84:6789/0 conn(0x7fe5280b17d0 0x7fe5280b01a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.84:6789/0 says I am v1:172.21.15.84:39542/0 (socket says 172.21.15.84:39542) 2024-03-29T11:27:03.444 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:03.447+0000 7fe52659c640 1 -- 172.21.15.84:0/2596889841 learned_addr learned my addr 172.21.15.84:0/2596889841 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-29T11:27:03.445 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:03.447+0000 7fe52c863640 10 monclient(hunting): picked mon.noname-c con 0x7fe528142690 addr v1:172.21.15.84:6790/0 2024-03-29T11:27:03.445 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:03.447+0000 7fe52c863640 1 -- 172.21.15.84:0/2596889841 --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe52813c570 con 0x7fe5280b17d0 2024-03-29T11:27:03.445 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:03.447+0000 7fe52c863640 1 -- 172.21.15.84:0/2596889841 --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe5280af9d0 con 0x7fe5281422b0 2024-03-29T11:27:03.445 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:03.447+0000 7fe52c863640 1 -- 172.21.15.84:0/2596889841 --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe5280afca0 con 0x7fe528142690 2024-03-29T11:27:03.445 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:03.447+0000 7fe52c863640 10 monclient(hunting): _renew_subs 2024-03-29T11:27:03.445 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:03.447+0000 7fe52c863640 10 monclient(hunting): authenticate will time out at 1284.741705s 2024-03-29T11:27:03.445 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:03.447+0000 7fe51effd640 10 client.?.objecter ms_handle_connect 0x7fe5281422b0 2024-03-29T11:27:03.445 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:03.447+0000 7fe51effd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:27:03.445 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:03.447+0000 7fe51effd640 10 client.?.objecter ms_handle_connect 0x7fe5280b17d0 2024-03-29T11:27:03.445 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:03.447+0000 7fe51effd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:27:03.445 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:03.447+0000 7fe51effd640 10 client.?.objecter ms_handle_connect 0x7fe528142690 2024-03-29T11:27:03.445 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:03.447+0000 7fe51effd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:27:03.445 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:03.447+0000 7fe51effd640 1 -- 172.21.15.84:0/2596889841 <== mon.1 v1:172.21.15.179:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3615798561 0 0) 0x7fe5280af9d0 con 0x7fe5281422b0 2024-03-29T11:27:03.445 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:03.447+0000 7fe51effd640 10 monclient(hunting): _negotiate 2024-03-29T11:27:03.446 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:03.447+0000 7fe51effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:27:03.446 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:03.447+0000 7fe51effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:27:03.446 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:03.447+0000 7fe51effd640 10 monclient(hunting): my global_id is 4450 2024-03-29T11:27:03.446 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:03.447+0000 7fe51effd640 1 -- 172.21.15.84:0/2596889841 --> v1:172.21.15.179:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe510004eb0 con 0x7fe5281422b0 2024-03-29T11:27:03.446 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:03.447+0000 7fe51effd640 1 -- 172.21.15.84:0/2596889841 <== mon.0 v1:172.21.15.84:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 638274864 0 0) 0x7fe52813c570 con 0x7fe5280b17d0 2024-03-29T11:27:03.446 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:03.447+0000 7fe51effd640 10 monclient(hunting): _negotiate 2024-03-29T11:27:03.446 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:03.447+0000 7fe51effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:27:03.446 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:03.447+0000 7fe51effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:27:03.446 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:03.447+0000 7fe51effd640 10 monclient(hunting): my global_id is 4509 2024-03-29T11:27:03.446 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:03.447+0000 7fe51effd640 1 -- 172.21.15.84:0/2596889841 --> v1:172.21.15.84:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe510006230 con 0x7fe5280b17d0 2024-03-29T11:27:03.446 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:03.447+0000 7fe51effd640 1 -- 172.21.15.84:0/2596889841 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1385069518 0 0) 0x7fe5280afca0 con 0x7fe528142690 2024-03-29T11:27:03.446 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:03.447+0000 7fe51effd640 10 monclient(hunting): _negotiate 2024-03-29T11:27:03.446 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:03.447+0000 7fe51effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:27:03.446 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:03.447+0000 7fe51effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:27:03.447 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:03.447+0000 7fe51effd640 10 monclient(hunting): my global_id is 4529 2024-03-29T11:27:03.447 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:03.447+0000 7fe51effd640 1 -- 172.21.15.84:0/2596889841 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe52813c570 con 0x7fe528142690 2024-03-29T11:27:03.447 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:03.448+0000 7fe51effd640 1 -- 172.21.15.84:0/2596889841 <== mon.2 v1:172.21.15.84:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 643827365 0 0) 0x7fe52813c570 con 0x7fe528142690 2024-03-29T11:27:03.447 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:03.448+0000 7fe51effd640 1 -- 172.21.15.84:0/2596889841 --> v1:172.21.15.84:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe5280afca0 con 0x7fe528142690 2024-03-29T11:27:03.447 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:03.448+0000 7fe51effd640 1 -- 172.21.15.84:0/2596889841 <== mon.2 v1:172.21.15.84:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4236868694 0 0) 0x7fe520001b60 con 0x7fe528142690 2024-03-29T11:27:03.447 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:03.448+0000 7fe51effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:27:03.447 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:03.448+0000 7fe51effd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:27:03.447 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:03.448+0000 7fe51effd640 10 monclient(hunting): dump: 2024-03-29T11:27:03.447 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T11:27:03.447 INFO:teuthology.misc.health.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:27:03.447 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T11:18:53.864025+0000 2024-03-29T11:27:03.447 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T11:18:53.864025+0000 2024-03-29T11:27:03.447 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T11:27:03.447 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:27:03.447 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T11:27:03.448 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T11:27:03.448 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T11:27:03.448 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:27:03.448 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:03.448+0000 7fe51effd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:27:03.448 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:03.448+0000 7fe51effd640 1 -- 172.21.15.84:0/2596889841 <== mon.0 v1:172.21.15.84:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 681002158 0 0) 0x7fe510006230 con 0x7fe5280b17d0 2024-03-29T11:27:03.448 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:03.448+0000 7fe51effd640 1 -- 172.21.15.84:0/2596889841 --> v1:172.21.15.84:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe52813c570 con 0x7fe5280b17d0 2024-03-29T11:27:03.448 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:03.448+0000 7fe51effd640 1 -- 172.21.15.84:0/2596889841 <== mon.0 v1:172.21.15.84:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4236868694 0 0) 0x7fe518001b00 con 0x7fe5280b17d0 2024-03-29T11:27:03.448 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:03.448+0000 7fe51effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:27:03.448 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:03.448+0000 7fe51effd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:27:03.448 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:03.448+0000 7fe51effd640 10 monclient(hunting): dump: 2024-03-29T11:27:03.448 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T11:27:03.448 INFO:teuthology.misc.health.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:27:03.448 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T11:18:53.864025+0000 2024-03-29T11:27:03.448 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T11:18:53.864025+0000 2024-03-29T11:27:03.448 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T11:27:03.449 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:27:03.449 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T11:27:03.449 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T11:27:03.449 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T11:27:03.449 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:27:03.449 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:03.449+0000 7fe51effd640 1 -- 172.21.15.84:0/2596889841 <== mon.0 v1:172.21.15.84:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2482222248 0 0) 0x7fe52813c570 con 0x7fe5280b17d0 2024-03-29T11:27:03.449 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:03.449+0000 7fe51effd640 1 -- 172.21.15.84:0/2596889841 >> v1:172.21.15.84:6790/0 conn(0x7fe528142690 legacy=0x7fe5280af2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:03.449 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:03.449+0000 7fe51effd640 1 -- 172.21.15.84:0/2596889841 >> v1:172.21.15.179:6789/0 conn(0x7fe5281422b0 legacy=0x7fe5281d67e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:03.449 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:03.449+0000 7fe51effd640 10 monclient: _finish_hunting 0 2024-03-29T11:27:03.449 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:03.449+0000 7fe51effd640 1 monclient: found mon.a 2024-03-29T11:27:03.449 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:03.449+0000 7fe51effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:27:03.449 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:03.449+0000 7fe51effd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.84:6789/0 2024-03-29T11:27:03.449 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:03.449+0000 7fe51effd640 1 -- 172.21.15.84:0/2596889841 --> v1:172.21.15.84:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5280b1440 con 0x7fe5280b17d0 2024-03-29T11:27:03.449 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:03.449+0000 7fe51effd640 10 monclient: _finish_auth 0 2024-03-29T11:27:03.449 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:03.449+0000 7fe51effd640 10 monclient: _check_auth_tickets 2024-03-29T11:27:03.450 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:03.449+0000 7fe51effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:27:03.450 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:03.449+0000 7fe52c863640 5 monclient: authenticate success, global_id 4509 2024-03-29T11:27:03.450 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:03.449+0000 7fe52c863640 10 monclient: _renew_subs 2024-03-29T11:27:03.450 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:03.449+0000 7fe52c863640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.84:6789/0 2024-03-29T11:27:03.450 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:03.449+0000 7fe52c863640 1 -- 172.21.15.84:0/2596889841 --> v1:172.21.15.84:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe5280adcf0 con 0x7fe5280b17d0 2024-03-29T11:27:03.450 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:03.449+0000 7fe52c863640 10 client.4509.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:27:03.450 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:03.449+0000 7fe52c863640 10 monclient: _renew_subs 2024-03-29T11:27:03.450 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:03.449+0000 7fe52c863640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.84:6789/0 2024-03-29T11:27:03.450 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:03.449+0000 7fe52c863640 1 -- 172.21.15.84:0/2596889841 --> v1:172.21.15.84:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe5281e2f90 con 0x7fe5280b17d0 2024-03-29T11:27:03.450 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:03.449+0000 7fe52c863640 1 librados: init done 2024-03-29T11:27:03.450 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:03.450+0000 7fe51effd640 1 -- 172.21.15.84:0/2596889841 <== mon.0 v1:172.21.15.84:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe5180010e0 con 0x7fe5280b17d0 2024-03-29T11:27:03.451 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:03.450+0000 7fe51effd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:27:03.451 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:03.450+0000 7fe51effd640 1 -- 172.21.15.84:0/2596889841 <== mon.0 v1:172.21.15.84:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4236868694 0 0) 0x7fe518001b00 con 0x7fe5280b17d0 2024-03-29T11:27:03.451 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:03.450+0000 7fe51effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:27:03.451 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:03.450+0000 7fe51effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:27:03.451 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:03.450+0000 7fe51effd640 10 monclient: dump: 2024-03-29T11:27:03.451 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T11:27:03.451 INFO:teuthology.misc.health.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:27:03.451 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T11:18:53.864025+0000 2024-03-29T11:27:03.451 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T11:18:53.864025+0000 2024-03-29T11:27:03.451 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T11:27:03.451 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:27:03.452 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T11:27:03.452 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T11:27:03.452 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T11:27:03.452 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:27:03.452 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:03.450+0000 7fe51ffff640 4 set_mon_vals no callback set 2024-03-29T11:27:03.452 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:03.450+0000 7fe52c863640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:27:03.452 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:03.451+0000 7fe52c863640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:27:03.452 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:03.451+0000 7fe52c863640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.84:6789/0 2024-03-29T11:27:03.452 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:03.451+0000 7fe52c863640 1 -- 172.21.15.84:0/2596889841 --> v1:172.21.15.84:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe504005080 con 0x7fe5280b17d0 2024-03-29T11:27:03.452 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:03.452+0000 7fe51effd640 1 -- 172.21.15.84:0/2596889841 <== mon.0 v1:172.21.15.84:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4033942909 0 0) 0x7fe51801a900 con 0x7fe5280b17d0 2024-03-29T11:27:03.452 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:03.452+0000 7fe51effd640 1 -- 172.21.15.84:0/2596889841 <== mon.0 v1:172.21.15.84:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2808139587 0 0) 0x7fe518084730 con 0x7fe5280b17d0 2024-03-29T11:27:03.452 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:03.452+0000 7fe51effd640 10 client.4509.objecter ms_dispatch 0x7fe528000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:27:03.452 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:03.452+0000 7fe51effd640 3 client.4509.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:27:03.452 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:03.452+0000 7fe51effd640 3 client.4509.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:27:03.452 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:03.452+0000 7fe51effd640 20 client.4509.objecter dump_active .. 0 homeless 2024-03-29T11:27:03.453 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:03.452+0000 7fe525d9b640 10 monclient: get_auth_request con 0x7fe51006e3c0 auth_method 0 2024-03-29T11:27:03.453 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:03.453+0000 7fe51effd640 10 client.4509.objecter ms_handle_connect 0x7fe51006e3c0 2024-03-29T11:27:03.453 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:03.461+0000 7fe51effd640 1 -- 172.21.15.84:0/2596889841 <== mon.0 v1:172.21.15.84:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fe518050070 con 0x7fe5280b17d0 2024-03-29T11:27:03.453 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:03.461+0000 7fe51effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:27:03.453 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:03.461+0000 7fe51effd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:27:03.786 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:03.798+0000 7fe52c863640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:27:03.787 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:03.798+0000 7fe52c863640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:27:03.787 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:03.798+0000 7fe52c863640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.84:6789/0 2024-03-29T11:27:03.787 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:03.799+0000 7fe52c863640 1 -- 172.21.15.84:0/2596889841 --> v1:172.21.15.84:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe504005330 con 0x7fe5280b17d0 2024-03-29T11:27:03.787 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:03.800+0000 7fe51effd640 1 -- 172.21.15.84:0/2596889841 <== mon.0 v1:172.21.15.84:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4209477549) 0x7fe518050070 con 0x7fe5280b17d0 2024-03-29T11:27:03.787 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:03.800+0000 7fe51effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:27:03.787 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:03.800+0000 7fe51effd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:27:03.788 INFO:teuthology.misc.health.smithi084.stdout: 2024-03-29T11:27:03.788 INFO:teuthology.misc.health.smithi084.stdout:{"epoch":10,"fsid":"2c3e002f-9eb0-4860-853b-4ef1ef562213","created":"2024-03-29T11:19:32.029169+0000","modified":"2024-03-29T11:19:34.200299+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:34.200299+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"5c39e276-2034-4716-bbc0-57a634eefa92","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"cd580e99-109f-4bf1-88c0-a5987bc8b9ed","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"11e60a5a-051d-453f-a190-66387cb35592","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"2c329fbe-bbc4-4030-bf87-d711cad04679","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f4213d4a-bd36-4433-91ae-d7e2b02481d3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"f0cf44ea-6055-48af-805b-aed858102ff9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"d895eaa0-cc0b-4ee4-9b5a-2dfcd89a60b7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"9bc5a7bc-8f52-4db9-96d6-de5e68d52eaa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:27:03.791 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:03.803+0000 7fe52c863640 10 librados: watch_flush enter 2024-03-29T11:27:03.791 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:03.803+0000 7fe52c863640 10 librados: watch_flush exit 2024-03-29T11:27:03.791 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:03.803+0000 7fe52c863640 20 client.4509.objecter shutdown clearing up homeless session... 2024-03-29T11:27:03.791 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:03.803+0000 7fe52c863640 10 client.4509.objecter successfully canceled tick 2024-03-29T11:27:03.791 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:03.803+0000 7fe52c863640 1 -- 172.21.15.84:0/2596889841 >> v1:172.21.15.179:6816/2981134980 conn(0x7fe51006e3c0 legacy=0x7fe510070880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:03.791 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:03.803+0000 7fe52c863640 10 monclient: shutdown 2024-03-29T11:27:03.791 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:03.803+0000 7fe52c863640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:27:03.791 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:03.803+0000 7fe52c863640 1 -- 172.21.15.84:0/2596889841 >> v1:172.21.15.84:6789/0 conn(0x7fe5280b17d0 legacy=0x7fe5280b01a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:03.791 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:03.804+0000 7fe52c863640 1 -- 172.21.15.84:0/2596889841 shutdown_connections 2024-03-29T11:27:03.792 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:03.804+0000 7fe52c863640 1 -- 172.21.15.84:0/2596889841 >> 172.21.15.84:0/2596889841 conn(0x7fe528066a00 msgr2=0x7fe52813d9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:27:03.792 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:03.804+0000 7fe52c863640 1 -- 172.21.15.84:0/2596889841 shutdown_connections 2024-03-29T11:27:03.792 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:03.804+0000 7fe52c863640 1 -- 172.21.15.84:0/2596889841 wait complete. 2024-03-29T11:27:03.792 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:03.804+0000 7fe52c863640 1 librados: shutdown 2024-03-29T11:27:03.806 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:27:10.612 DEBUG:teuthology.orchestra.run.smithi084:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:27:10.770 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:10.781+0000 7f86e8a0f640 10 monclient: get_monmap_and_config 2024-03-29T11:27:10.770 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:10.781+0000 7f86e8a0f640 10 monclient: build_initial_monmap 2024-03-29T11:27:10.770 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:10.782+0000 7f86e8a0f640 10 monclient: monmap: 2024-03-29T11:27:10.770 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-29T11:27:10.770 INFO:teuthology.misc.health.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:27:10.770 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T11:27:10.782915+0000 2024-03-29T11:27:10.771 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T11:27:10.782915+0000 2024-03-29T11:27:10.771 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-29T11:27:10.771 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:27:10.771 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-29T11:27:10.771 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.noname-c 2024-03-29T11:27:10.771 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-b 2024-03-29T11:27:10.771 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:27:10.771 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:10.783+0000 7f86e8a0f640 1 Processor -- start 2024-03-29T11:27:10.772 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:10.783+0000 7f86e8a0f640 1 -- start start 2024-03-29T11:27:10.773 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:10.783+0000 7f86e8a0f640 10 monclient: init 2024-03-29T11:27:10.773 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:10.784+0000 7f86e8a0f640 10 monclient: _reopen_session rank -1 2024-03-29T11:27:10.773 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:10.784+0000 7f86e8a0f640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:27:10.773 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:10.784+0000 7f86e8a0f640 10 monclient(hunting): picked mon.noname-c con 0x7f86e40a9ba0 addr v1:172.21.15.179:6789/0 2024-03-29T11:27:10.773 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:10.784+0000 7f86e8a0f640 10 monclient(hunting): picked mon.noname-a con 0x7f86e413b5a0 addr v1:172.21.15.84:6789/0 2024-03-29T11:27:10.773 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:10.784+0000 7f86e8a0f640 10 monclient(hunting): picked mon.noname-b con 0x7f86e40ab550 addr v1:172.21.15.84:6790/0 2024-03-29T11:27:10.773 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:10.784+0000 7f86e8a0f640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f86e40ac430 con 0x7f86e413b5a0 2024-03-29T11:27:10.773 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:10.784+0000 7f86e8a0f640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f86e40ac680 con 0x7f86e40a9ba0 2024-03-29T11:27:10.773 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:10.784+0000 7f86e8a0f640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f86e40ac960 con 0x7f86e40ab550 2024-03-29T11:27:10.773 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:10.784+0000 7f86e8a0f640 10 monclient(hunting): _renew_subs 2024-03-29T11:27:10.773 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:10.784+0000 7f86e8a0f640 10 monclient(hunting): authenticate will time out at 1292.078919s 2024-03-29T11:27:10.774 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:10.787+0000 7f86e2d9d640 1 --1- >> v1:172.21.15.84:6790/0 conn(0x7f86e40ab550 0x7f86e41498d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.84:6790/0 says I am v1:172.21.15.84:46724/0 (socket says 172.21.15.84:46724) 2024-03-29T11:27:10.774 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:10.787+0000 7f86e2d9d640 1 -- 172.21.15.84:0/2948416128 learned_addr learned my addr 172.21.15.84:0/2948416128 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-29T11:27:10.780 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:10.788+0000 7f86e159a640 1 -- 172.21.15.84:0/2948416128 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2653002967 0 0) 0x7f86e40ac960 con 0x7f86e40ab550 2024-03-29T11:27:10.780 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:10.788+0000 7f86e159a640 10 monclient(hunting): _negotiate 2024-03-29T11:27:10.780 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:10.788+0000 7f86e159a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:27:10.780 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:10.788+0000 7f86e159a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:27:10.780 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:10.788+0000 7f86e159a640 10 monclient(hunting): my global_id is 4532 2024-03-29T11:27:10.780 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:10.788+0000 7f86e159a640 1 -- 172.21.15.84:0/2948416128 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f86cc003520 con 0x7f86e40ab550 2024-03-29T11:27:10.780 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:10.788+0000 7f86e259c640 1 -- 172.21.15.84:0/2948416128 >> v1:172.21.15.179:6789/0 conn(0x7f86e40a9ba0 legacy=0x7f86e413ae90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-03-29T11:27:10.781 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:10.788+0000 7f86e259c640 1 -- 172.21.15.84:0/2948416128 >> v1:172.21.15.179:6789/0 conn(0x7f86e40a9ba0 legacy=0x7f86e413ae90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-29T11:27:10.781 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:10.788+0000 7f86e259c640 1 --1- 172.21.15.84:0/2948416128 >> v1:172.21.15.179:6789/0 conn(0x7f86e40a9ba0 0x7f86e413ae90 :-1 s=OPENED pgs=925245 cs=1 l=1).handle_message read tag failed 2024-03-29T11:27:10.781 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:10.788+0000 7f86e259c640 1 --1- 172.21.15.84:0/2948416128 >> v1:172.21.15.179:6789/0 conn(0x7f86e40a9ba0 0x7f86e413ae90 :-1 s=OPENED pgs=925245 cs=1 l=1).fault on lossy channel, failing 2024-03-29T11:27:10.781 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:10.788+0000 7f86e159a640 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.179:6789/0 2024-03-29T11:27:10.781 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:10.789+0000 7f86e159a640 1 -- 172.21.15.84:0/2948416128 <== mon.2 v1:172.21.15.84:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2358936118 0 0) 0x7f86cc003520 con 0x7f86e40ab550 2024-03-29T11:27:10.781 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:10.789+0000 7f86e159a640 1 -- 172.21.15.84:0/2948416128 >> v1:172.21.15.179:6789/0 conn(0x7f86e40a9ba0 legacy=0x7f86e413ae90 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-29T11:27:10.781 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:10.789+0000 7f86e159a640 1 -- 172.21.15.84:0/2948416128 >> v1:172.21.15.84:6789/0 conn(0x7f86e413b5a0 legacy=0x7f86e40aae40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:10.781 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:10.789+0000 7f86e159a640 10 monclient: _finish_hunting 0 2024-03-29T11:27:10.781 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:10.789+0000 7f86e159a640 1 monclient: found mon.noname-b 2024-03-29T11:27:10.781 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:10.789+0000 7f86e159a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:27:10.781 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:10.789+0000 7f86e159a640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.84:6790/0 2024-03-29T11:27:10.781 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:10.789+0000 7f86e159a640 1 -- 172.21.15.84:0/2948416128 --> v1:172.21.15.84:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86e413baf0 con 0x7f86e40ab550 2024-03-29T11:27:10.781 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:10.789+0000 7f86e159a640 10 monclient: _finish_auth 0 2024-03-29T11:27:10.781 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:10.789+0000 7f86e159a640 10 monclient: _check_auth_tickets 2024-03-29T11:27:10.781 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:10.789+0000 7f86e159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:27:10.782 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:10.789+0000 7f86e159a640 1 -- 172.21.15.84:0/2948416128 <== mon.2 v1:172.21.15.84:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4236868694 0 0) 0x7f86d8004cb0 con 0x7f86e40ab550 2024-03-29T11:27:10.782 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:10.789+0000 7f86e159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:27:10.782 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:10.789+0000 7f86e159a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:27:10.782 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:10.789+0000 7f86e159a640 10 monclient: dump: 2024-03-29T11:27:10.782 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T11:27:10.782 INFO:teuthology.misc.health.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:27:10.782 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T11:18:53.864025+0000 2024-03-29T11:27:10.782 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T11:18:53.864025+0000 2024-03-29T11:27:10.782 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T11:27:10.782 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:27:10.782 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T11:27:10.783 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T11:27:10.783 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T11:27:10.783 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:27:10.783 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:10.789+0000 7f86e8a0f640 5 monclient: authenticate success, global_id 4532 2024-03-29T11:27:10.783 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:10.789+0000 7f86e8a0f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:27:10.783 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:10.792+0000 7f86e159a640 1 -- 172.21.15.84:0/2948416128 <== mon.2 v1:172.21.15.84:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f86d8004e10 con 0x7f86e40ab550 2024-03-29T11:27:10.783 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:10.792+0000 7f86e159a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:27:10.783 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:10.792+0000 7f86e8a0f640 10 monclient: get_monmap_and_config success 2024-03-29T11:27:10.783 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:10.792+0000 7f86e8a0f640 4 set_mon_vals no callback set 2024-03-29T11:27:10.783 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:10.793+0000 7f86e159a640 1 -- 172.21.15.84:0/2948416128 <== mon.2 v1:172.21.15.84:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4236868694 0 0) 0x7f86d80013f0 con 0x7f86e40ab550 2024-03-29T11:27:10.783 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:10.793+0000 7f86e159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:27:10.783 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:10.793+0000 7f86e159a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:27:10.783 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:10.793+0000 7f86e159a640 10 monclient: dump: 2024-03-29T11:27:10.783 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T11:27:10.783 INFO:teuthology.misc.health.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:27:10.784 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T11:18:53.864025+0000 2024-03-29T11:27:10.784 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T11:18:53.864025+0000 2024-03-29T11:27:10.784 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T11:27:10.784 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:27:10.784 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T11:27:10.784 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T11:27:10.784 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T11:27:10.784 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:27:10.785 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:10.793+0000 7f86e8a0f640 10 monclient: shutdown 2024-03-29T11:27:10.785 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:10.793+0000 7f86e8a0f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:27:10.785 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:10.793+0000 7f86e8a0f640 1 -- 172.21.15.84:0/2948416128 >> v1:172.21.15.84:6790/0 conn(0x7f86e40ab550 legacy=0x7f86e41498d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:10.785 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:10.796+0000 7f86e8a0f640 1 -- 172.21.15.84:0/2948416128 shutdown_connections 2024-03-29T11:27:10.785 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:10.796+0000 7f86e8a0f640 1 -- 172.21.15.84:0/2948416128 >> 172.21.15.84:0/2948416128 conn(0x7f86e40b3a00 msgr2=0x7f86e40b3de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:27:10.785 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:10.797+0000 7f86e8a0f640 1 -- 172.21.15.84:0/2948416128 shutdown_connections 2024-03-29T11:27:10.785 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:10.797+0000 7f86e8a0f640 1 -- 172.21.15.84:0/2948416128 wait complete. 2024-03-29T11:27:10.789 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:10.798+0000 7f86e8a0f640 10 monclient: build_initial_monmap 2024-03-29T11:27:10.789 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:10.798+0000 7f86e8a0f640 10 monclient: monmap: 2024-03-29T11:27:10.789 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-29T11:27:10.789 INFO:teuthology.misc.health.smithi084.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:27:10.789 INFO:teuthology.misc.health.smithi084.stderr:last_changed 0.000000 2024-03-29T11:27:10.789 INFO:teuthology.misc.health.smithi084.stderr:created 0.000000 2024-03-29T11:27:10.789 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-29T11:27:10.789 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:27:10.789 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-29T11:27:10.789 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.noname-b 2024-03-29T11:27:10.789 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-c 2024-03-29T11:27:10.789 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:27:10.789 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:10.799+0000 7f86e8a0f640 1 librados: starting msgr at 2024-03-29T11:27:10.790 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:10.799+0000 7f86e8a0f640 1 librados: starting objecter 2024-03-29T11:27:10.790 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:10.799+0000 7f86e8a0f640 1 Processor -- start 2024-03-29T11:27:10.790 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:10.799+0000 7f86e8a0f640 1 -- start start 2024-03-29T11:27:10.790 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:10.799+0000 7f86e8a0f640 1 librados: setting wanted keys 2024-03-29T11:27:10.790 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:10.799+0000 7f86e8a0f640 1 librados: calling monclient init 2024-03-29T11:27:10.790 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:10.799+0000 7f86e8a0f640 10 monclient: init 2024-03-29T11:27:10.790 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:10.800+0000 7f86e8a0f640 10 monclient: _reopen_session rank -1 2024-03-29T11:27:10.790 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:10.800+0000 7f86e8a0f640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:27:10.791 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:10.803+0000 7f86e8a0f640 10 monclient(hunting): picked mon.noname-b con 0x7f86e40a9ba0 addr v1:172.21.15.179:6789/0 2024-03-29T11:27:10.791 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:10.803+0000 7f86e8a0f640 10 monclient(hunting): picked mon.noname-c con 0x7f86e40ab550 addr v1:172.21.15.84:6790/0 2024-03-29T11:27:10.792 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:10.803+0000 7f86e8a0f640 10 monclient(hunting): picked mon.noname-a con 0x7f86e413b5a0 addr v1:172.21.15.84:6789/0 2024-03-29T11:27:10.792 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:10.803+0000 7f86e8a0f640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f86e406fe80 con 0x7f86e413b5a0 2024-03-29T11:27:10.792 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:10.803+0000 7f86e8a0f640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f86e41d6260 con 0x7f86e40a9ba0 2024-03-29T11:27:10.792 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:10.803+0000 7f86e1d9b640 1 --1- >> v1:172.21.15.84:6790/0 conn(0x7f86e40ab550 0x7f86e406e370 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.84:6790/0 says I am v1:172.21.15.84:47576/0 (socket says 172.21.15.84:47576) 2024-03-29T11:27:10.792 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:10.803+0000 7f86e1d9b640 1 -- 172.21.15.84:0/409117623 learned_addr learned my addr 172.21.15.84:0/409117623 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-29T11:27:10.792 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:10.803+0000 7f86e8a0f640 1 -- 172.21.15.84:0/409117623 --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f86e4140380 con 0x7f86e40ab550 2024-03-29T11:27:10.792 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:10.803+0000 7f86e8a0f640 10 monclient(hunting): _renew_subs 2024-03-29T11:27:10.792 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:10.803+0000 7f86e8a0f640 10 monclient(hunting): authenticate will time out at 1292.097855s 2024-03-29T11:27:10.795 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:10.807+0000 7f86d2ffd640 10 client.?.objecter ms_handle_connect 0x7f86e40ab550 2024-03-29T11:27:10.796 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:10.807+0000 7f86d2ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:27:10.796 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:10.807+0000 7f86d2ffd640 10 client.?.objecter ms_handle_connect 0x7f86e40a9ba0 2024-03-29T11:27:10.796 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:10.807+0000 7f86d2ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:27:10.796 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:10.807+0000 7f86d2ffd640 1 -- 172.21.15.84:0/409117623 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2200053547 0 0) 0x7f86e4140380 con 0x7f86e40ab550 2024-03-29T11:27:10.796 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:10.807+0000 7f86d2ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:27:10.796 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:10.807+0000 7f86d2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:27:10.796 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:10.807+0000 7f86d2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:27:10.796 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:10.807+0000 7f86d2ffd640 10 monclient(hunting): my global_id is 4535 2024-03-29T11:27:10.796 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:10.807+0000 7f86d2ffd640 1 -- 172.21.15.84:0/409117623 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f86c4004eb0 con 0x7f86e40ab550 2024-03-29T11:27:10.797 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:10.807+0000 7f86d2ffd640 1 -- 172.21.15.84:0/409117623 <== mon.1 v1:172.21.15.179:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 60664214 0 0) 0x7f86e41d6260 con 0x7f86e40a9ba0 2024-03-29T11:27:10.797 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:10.807+0000 7f86d2ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:27:10.797 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:10.807+0000 7f86d2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:27:10.797 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:10.807+0000 7f86d2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:27:10.797 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:10.807+0000 7f86d2ffd640 10 monclient(hunting): my global_id is 4453 2024-03-29T11:27:10.797 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:10.807+0000 7f86d2ffd640 1 -- 172.21.15.84:0/409117623 --> v1:172.21.15.179:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f86e4140380 con 0x7f86e40a9ba0 2024-03-29T11:27:10.797 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:10.808+0000 7f86d2ffd640 10 client.?.objecter ms_handle_connect 0x7f86e413b5a0 2024-03-29T11:27:10.797 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:10.808+0000 7f86d2ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:27:10.799 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:10.808+0000 7f86d2ffd640 1 -- 172.21.15.84:0/409117623 <== mon.1 v1:172.21.15.179:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 431305729 0 0) 0x7f86e4140380 con 0x7f86e40a9ba0 2024-03-29T11:27:10.799 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:10.808+0000 7f86d2ffd640 1 -- 172.21.15.84:0/409117623 --> v1:172.21.15.179:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f86e41d6260 con 0x7f86e40a9ba0 2024-03-29T11:27:10.799 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:10.808+0000 7f86d2ffd640 1 -- 172.21.15.84:0/409117623 <== mon.1 v1:172.21.15.179:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4236868694 0 0) 0x7f86d40018a0 con 0x7f86e40a9ba0 2024-03-29T11:27:10.799 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:10.808+0000 7f86d2ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:27:10.799 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:10.808+0000 7f86d2ffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:27:10.799 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:10.808+0000 7f86d2ffd640 10 monclient(hunting): dump: 2024-03-29T11:27:10.799 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T11:27:10.799 INFO:teuthology.misc.health.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:27:10.800 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T11:18:53.864025+0000 2024-03-29T11:27:10.800 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T11:18:53.864025+0000 2024-03-29T11:27:10.800 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T11:27:10.800 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:27:10.800 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T11:27:10.800 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T11:27:10.800 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T11:27:10.800 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:27:10.801 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:10.808+0000 7f86d2ffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:27:10.801 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:10.808+0000 7f86d2ffd640 1 -- 172.21.15.84:0/409117623 <== mon.0 v1:172.21.15.84:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1430898905 0 0) 0x7f86e406fe80 con 0x7f86e413b5a0 2024-03-29T11:27:10.801 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:10.808+0000 7f86d2ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:27:10.801 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:10.808+0000 7f86d2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:27:10.801 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:10.808+0000 7f86d2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:27:10.801 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:10.808+0000 7f86d2ffd640 10 monclient(hunting): my global_id is 4512 2024-03-29T11:27:10.801 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:10.808+0000 7f86d2ffd640 1 -- 172.21.15.84:0/409117623 --> v1:172.21.15.84:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f86e4140380 con 0x7f86e413b5a0 2024-03-29T11:27:10.801 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:10.809+0000 7f86d2ffd640 1 -- 172.21.15.84:0/409117623 <== mon.1 v1:172.21.15.179:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 639128776 0 0) 0x7f86e41d6260 con 0x7f86e40a9ba0 2024-03-29T11:27:10.801 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:10.809+0000 7f86d2ffd640 1 -- 172.21.15.84:0/409117623 >> v1:172.21.15.84:6790/0 conn(0x7f86e40ab550 legacy=0x7f86e406e370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:10.801 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:10.809+0000 7f86d2ffd640 1 -- 172.21.15.84:0/409117623 >> v1:172.21.15.84:6789/0 conn(0x7f86e413b5a0 legacy=0x7f86e41dd740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:10.801 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:10.809+0000 7f86d2ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:27:10.801 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:10.809+0000 7f86d2ffd640 1 monclient: found mon.b 2024-03-29T11:27:10.801 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:10.809+0000 7f86d2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:27:10.801 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:10.809+0000 7f86d2ffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-29T11:27:10.802 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:10.809+0000 7f86d2ffd640 1 -- 172.21.15.84:0/409117623 --> v1:172.21.15.179:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86e4140510 con 0x7f86e40a9ba0 2024-03-29T11:27:10.802 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:10.809+0000 7f86d2ffd640 10 monclient: _finish_auth 0 2024-03-29T11:27:10.802 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:10.809+0000 7f86d2ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:27:10.802 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:10.809+0000 7f86d2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:27:10.802 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:10.809+0000 7f86e8a0f640 5 monclient: authenticate success, global_id 4453 2024-03-29T11:27:10.802 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:10.809+0000 7f86e8a0f640 10 monclient: _renew_subs 2024-03-29T11:27:10.802 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:10.809+0000 7f86e8a0f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-29T11:27:10.802 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:10.809+0000 7f86e8a0f640 1 -- 172.21.15.84:0/409117623 --> v1:172.21.15.179:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f86e41d6e90 con 0x7f86e40a9ba0 2024-03-29T11:27:10.802 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:10.809+0000 7f86e8a0f640 10 client.4453.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:27:10.802 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:10.809+0000 7f86e8a0f640 10 monclient: _renew_subs 2024-03-29T11:27:10.802 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:10.809+0000 7f86d2ffd640 1 -- 172.21.15.84:0/409117623 <== mon.1 v1:172.21.15.179:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f86d4001a00 con 0x7f86e40a9ba0 2024-03-29T11:27:10.802 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:10.809+0000 7f86e8a0f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-29T11:27:10.802 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:10.809+0000 7f86e8a0f640 1 -- 172.21.15.84:0/409117623 --> v1:172.21.15.179:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f86e406ec60 con 0x7f86e40a9ba0 2024-03-29T11:27:10.802 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:10.809+0000 7f86e8a0f640 1 librados: init done 2024-03-29T11:27:10.802 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:10.809+0000 7f86d2ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:27:10.803 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:10.809+0000 7f86d2ffd640 1 -- 172.21.15.84:0/409117623 <== mon.1 v1:172.21.15.179:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4236868694 0 0) 0x7f86d40037d0 con 0x7f86e40a9ba0 2024-03-29T11:27:10.803 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:10.809+0000 7f86d2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:27:10.803 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:10.809+0000 7f86d2ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:27:10.803 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:10.809+0000 7f86d2ffd640 10 monclient: dump: 2024-03-29T11:27:10.803 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T11:27:10.803 INFO:teuthology.misc.health.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:27:10.803 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T11:18:53.864025+0000 2024-03-29T11:27:10.803 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T11:18:53.864025+0000 2024-03-29T11:27:10.803 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T11:27:10.803 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:27:10.803 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T11:27:10.803 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T11:27:10.803 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T11:27:10.803 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:27:10.803 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:10.809+0000 7f86e0d99640 4 set_mon_vals no callback set 2024-03-29T11:27:10.804 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:10.811+0000 7f86d2ffd640 1 -- 172.21.15.84:0/409117623 <== mon.1 v1:172.21.15.179:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4033942909 0 0) 0x7f86d4003930 con 0x7f86e40a9ba0 2024-03-29T11:27:10.804 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:10.812+0000 7f86e8a0f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:27:10.804 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:10.812+0000 7f86e8a0f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:27:10.804 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:10.812+0000 7f86d2ffd640 1 -- 172.21.15.84:0/409117623 <== mon.1 v1:172.21.15.179:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2808139587 0 0) 0x7f86d40849b0 con 0x7f86e40a9ba0 2024-03-29T11:27:10.804 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:10.812+0000 7f86d2ffd640 10 client.4453.objecter ms_dispatch 0x7f86e4000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:27:10.804 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:10.812+0000 7f86d2ffd640 3 client.4453.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:27:10.804 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:10.812+0000 7f86d2ffd640 3 client.4453.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:27:10.804 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:10.812+0000 7f86d2ffd640 20 client.4453.objecter dump_active .. 0 homeless 2024-03-29T11:27:10.805 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:10.812+0000 7f86e8a0f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-29T11:27:10.805 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:10.812+0000 7f86e8a0f640 1 -- 172.21.15.84:0/409117623 --> v1:172.21.15.179:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f86c8005080 con 0x7f86e40a9ba0 2024-03-29T11:27:10.805 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:10.812+0000 7f86e1d9b640 10 monclient: get_auth_request con 0x7f86c406dd30 auth_method 0 2024-03-29T11:27:10.805 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:10.813+0000 7f86d2ffd640 10 client.4453.objecter ms_handle_connect 0x7f86c406dd30 2024-03-29T11:27:10.806 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:10.819+0000 7f86d2ffd640 1 -- 172.21.15.84:0/409117623 <== mon.1 v1:172.21.15.179:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f86d40500c0 con 0x7f86e40a9ba0 2024-03-29T11:27:10.806 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:10.819+0000 7f86d2ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:27:10.806 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:10.819+0000 7f86d2ffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:27:11.181 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:11.194+0000 7f86e8a0f640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:27:11.182 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:11.194+0000 7f86e8a0f640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:27:11.182 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:11.194+0000 7f86e8a0f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-29T11:27:11.182 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:11.194+0000 7f86e8a0f640 1 -- 172.21.15.84:0/409117623 --> v1:172.21.15.179:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f86c8005330 con 0x7f86e40a9ba0 2024-03-29T11:27:11.183 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:11.195+0000 7f86d2ffd640 1 -- 172.21.15.84:0/409117623 <== mon.1 v1:172.21.15.179:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4209477549) 0x7f86d40521c0 con 0x7f86e40a9ba0 2024-03-29T11:27:11.183 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:11.195+0000 7f86d2ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:27:11.183 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:11.195+0000 7f86d2ffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:27:11.183 INFO:teuthology.misc.health.smithi084.stdout: 2024-03-29T11:27:11.183 INFO:teuthology.misc.health.smithi084.stdout:{"epoch":10,"fsid":"2c3e002f-9eb0-4860-853b-4ef1ef562213","created":"2024-03-29T11:19:32.029169+0000","modified":"2024-03-29T11:19:34.200299+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:34.200299+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"5c39e276-2034-4716-bbc0-57a634eefa92","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"cd580e99-109f-4bf1-88c0-a5987bc8b9ed","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"11e60a5a-051d-453f-a190-66387cb35592","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"2c329fbe-bbc4-4030-bf87-d711cad04679","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f4213d4a-bd36-4433-91ae-d7e2b02481d3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"f0cf44ea-6055-48af-805b-aed858102ff9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"d895eaa0-cc0b-4ee4-9b5a-2dfcd89a60b7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"9bc5a7bc-8f52-4db9-96d6-de5e68d52eaa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:27:11.186 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:11.198+0000 7f86e8a0f640 10 librados: watch_flush enter 2024-03-29T11:27:11.186 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:11.199+0000 7f86e8a0f640 10 librados: watch_flush exit 2024-03-29T11:27:11.187 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:11.199+0000 7f86e8a0f640 20 client.4453.objecter shutdown clearing up homeless session... 2024-03-29T11:27:11.187 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:11.199+0000 7f86e8a0f640 10 client.4453.objecter successfully canceled tick 2024-03-29T11:27:11.187 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:11.199+0000 7f86e8a0f640 1 -- 172.21.15.84:0/409117623 >> v1:172.21.15.179:6816/2981134980 conn(0x7f86c406dd30 legacy=0x7f86c40701f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:11.187 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:11.199+0000 7f86e8a0f640 10 monclient: shutdown 2024-03-29T11:27:11.187 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:11.199+0000 7f86e8a0f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:27:11.187 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:11.199+0000 7f86e8a0f640 1 -- 172.21.15.84:0/409117623 >> v1:172.21.15.179:6789/0 conn(0x7f86e40a9ba0 legacy=0x7f86e41d6780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:11.187 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:11.199+0000 7f86e8a0f640 1 -- 172.21.15.84:0/409117623 shutdown_connections 2024-03-29T11:27:11.187 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:11.199+0000 7f86e8a0f640 1 -- 172.21.15.84:0/409117623 >> 172.21.15.84:0/409117623 conn(0x7f86e40b3a00 msgr2=0x7f86e4136e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:27:11.187 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:11.199+0000 7f86e8a0f640 1 -- 172.21.15.84:0/409117623 shutdown_connections 2024-03-29T11:27:11.187 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:11.199+0000 7f86e8a0f640 1 -- 172.21.15.84:0/409117623 wait complete. 2024-03-29T11:27:11.187 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:11.199+0000 7f86e8a0f640 1 librados: shutdown 2024-03-29T11:27:11.202 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:27:18.010 DEBUG:teuthology.orchestra.run.smithi084:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:27:18.160 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:18.170+0000 7f1df904a640 10 monclient: get_monmap_and_config 2024-03-29T11:27:18.160 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:18.170+0000 7f1df904a640 10 monclient: build_initial_monmap 2024-03-29T11:27:18.160 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:18.171+0000 7f1df904a640 10 monclient: monmap: 2024-03-29T11:27:18.160 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-29T11:27:18.161 INFO:teuthology.misc.health.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:27:18.161 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T11:27:18.172902+0000 2024-03-29T11:27:18.161 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T11:27:18.172902+0000 2024-03-29T11:27:18.161 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-29T11:27:18.161 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:27:18.161 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-29T11:27:18.161 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.noname-c 2024-03-29T11:27:18.161 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-b 2024-03-29T11:27:18.161 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:27:18.161 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:18.172+0000 7f1df904a640 1 Processor -- start 2024-03-29T11:27:18.162 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:18.172+0000 7f1df904a640 1 -- start start 2024-03-29T11:27:18.162 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:18.172+0000 7f1df904a640 10 monclient: init 2024-03-29T11:27:18.162 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:18.172+0000 7f1df904a640 10 monclient: _reopen_session rank -1 2024-03-29T11:27:18.163 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:18.172+0000 7f1df904a640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:27:18.163 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:18.173+0000 7f1df904a640 10 monclient(hunting): picked mon.noname-c con 0x7f1df40a9ac0 addr v1:172.21.15.179:6789/0 2024-03-29T11:27:18.163 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:18.173+0000 7f1df904a640 10 monclient(hunting): picked mon.noname-a con 0x7f1df413ee90 addr v1:172.21.15.84:6789/0 2024-03-29T11:27:18.163 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:18.173+0000 7f1df904a640 10 monclient(hunting): picked mon.noname-b con 0x7f1df40af900 addr v1:172.21.15.84:6790/0 2024-03-29T11:27:18.163 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:18.173+0000 7f1df904a640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1df40afd10 con 0x7f1df413ee90 2024-03-29T11:27:18.163 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:18.173+0000 7f1df904a640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1df40af6f0 con 0x7f1df40a9ac0 2024-03-29T11:27:18.163 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:18.173+0000 7f1df904a640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1df40aac80 con 0x7f1df40af900 2024-03-29T11:27:18.163 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:18.173+0000 7f1df904a640 10 monclient(hunting): _renew_subs 2024-03-29T11:27:18.163 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:18.173+0000 7f1df904a640 10 monclient(hunting): authenticate will time out at 1299.469092s 2024-03-29T11:27:18.165 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:18.174+0000 7f1df2d9d640 1 --1- >> v1:172.21.15.179:6789/0 conn(0x7f1df40a9ac0 0x7f1df40adfe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.179:6789/0 says I am v1:172.21.15.84:47274/0 (socket says 172.21.15.84:47274) 2024-03-29T11:27:18.165 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:18.174+0000 7f1df2d9d640 1 -- 172.21.15.84:0/2184344832 learned_addr learned my addr 172.21.15.84:0/2184344832 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-29T11:27:18.165 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:18.175+0000 7f1df1d9b640 1 -- 172.21.15.84:0/2184344832 <== mon.1 v1:172.21.15.179:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 80970816 0 0) 0x7f1df40af6f0 con 0x7f1df40a9ac0 2024-03-29T11:27:18.165 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:18.175+0000 7f1df1d9b640 10 monclient(hunting): _negotiate 2024-03-29T11:27:18.165 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:18.175+0000 7f1df1d9b640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:27:18.165 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:18.175+0000 7f1df1d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:27:18.165 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:18.175+0000 7f1df1d9b640 10 monclient(hunting): my global_id is 4456 2024-03-29T11:27:18.165 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:18.175+0000 7f1df1d9b640 1 -- 172.21.15.84:0/2184344832 --> v1:172.21.15.179:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1ddc003520 con 0x7f1df40a9ac0 2024-03-29T11:27:18.165 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:18.175+0000 7f1df1d9b640 1 -- 172.21.15.84:0/2184344832 <== mon.1 v1:172.21.15.179:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3967994274 0 0) 0x7f1ddc003520 con 0x7f1df40a9ac0 2024-03-29T11:27:18.165 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:18.175+0000 7f1df1d9b640 1 -- 172.21.15.84:0/2184344832 >> v1:172.21.15.84:6790/0 conn(0x7f1df40af900 legacy=0x7f1df4149b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:18.165 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:18.175+0000 7f1df1d9b640 1 -- 172.21.15.84:0/2184344832 >> v1:172.21.15.84:6789/0 conn(0x7f1df413ee90 legacy=0x7f1df413b5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:18.166 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:18.175+0000 7f1df1d9b640 10 monclient: _finish_hunting 0 2024-03-29T11:27:18.166 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:18.175+0000 7f1df1d9b640 1 monclient: found mon.noname-c 2024-03-29T11:27:18.166 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:18.175+0000 7f1df1d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:27:18.166 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:18.175+0000 7f1df1d9b640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.179:6789/0 2024-03-29T11:27:18.166 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:18.175+0000 7f1df1d9b640 1 -- 172.21.15.84:0/2184344832 --> v1:172.21.15.179:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1df40aaf60 con 0x7f1df40a9ac0 2024-03-29T11:27:18.166 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:18.175+0000 7f1df1d9b640 10 monclient: _finish_auth 0 2024-03-29T11:27:18.166 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:18.175+0000 7f1df1d9b640 10 monclient: _check_auth_tickets 2024-03-29T11:27:18.166 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:18.175+0000 7f1df1d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:27:18.166 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:18.175+0000 7f1df1d9b640 1 -- 172.21.15.84:0/2184344832 <== mon.1 v1:172.21.15.179:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4236868694 0 0) 0x7f1de4001690 con 0x7f1df40a9ac0 2024-03-29T11:27:18.166 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:18.175+0000 7f1df904a640 5 monclient: authenticate success, global_id 4456 2024-03-29T11:27:18.166 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:18.175+0000 7f1df904a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:27:18.167 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:18.175+0000 7f1df1d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:27:18.167 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:18.175+0000 7f1df1d9b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:27:18.167 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:18.175+0000 7f1df1d9b640 10 monclient: dump: 2024-03-29T11:27:18.167 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T11:27:18.167 INFO:teuthology.misc.health.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:27:18.167 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T11:18:53.864025+0000 2024-03-29T11:27:18.167 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T11:18:53.864025+0000 2024-03-29T11:27:18.167 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T11:27:18.167 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:27:18.167 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T11:27:18.168 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T11:27:18.168 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T11:27:18.168 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:27:18.168 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:18.176+0000 7f1df904a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:27:18.168 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:18.176+0000 7f1df1d9b640 1 -- 172.21.15.84:0/2184344832 <== mon.1 v1:172.21.15.179:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1de40017f0 con 0x7f1df40a9ac0 2024-03-29T11:27:18.168 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:18.176+0000 7f1df1d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:27:18.168 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:18.176+0000 7f1df904a640 10 monclient: get_monmap_and_config success 2024-03-29T11:27:18.168 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:18.176+0000 7f1df1d9b640 1 -- 172.21.15.84:0/2184344832 <== mon.1 v1:172.21.15.179:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4236868694 0 0) 0x7f1de4001c00 con 0x7f1df40a9ac0 2024-03-29T11:27:18.168 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:18.176+0000 7f1df904a640 4 set_mon_vals no callback set 2024-03-29T11:27:18.168 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:18.176+0000 7f1df1d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:27:18.168 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:18.176+0000 7f1df1d9b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:27:18.169 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:18.176+0000 7f1df1d9b640 10 monclient: dump: 2024-03-29T11:27:18.169 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T11:27:18.169 INFO:teuthology.misc.health.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:27:18.169 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T11:18:53.864025+0000 2024-03-29T11:27:18.169 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T11:18:53.864025+0000 2024-03-29T11:27:18.169 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T11:27:18.169 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:27:18.169 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T11:27:18.169 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T11:27:18.169 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T11:27:18.169 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:27:18.171 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:18.177+0000 7f1df904a640 10 monclient: shutdown 2024-03-29T11:27:18.172 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:18.177+0000 7f1df904a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:27:18.172 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:18.177+0000 7f1df904a640 1 -- 172.21.15.84:0/2184344832 >> v1:172.21.15.179:6789/0 conn(0x7f1df40a9ac0 legacy=0x7f1df40adfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:18.172 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:18.177+0000 7f1df904a640 1 -- 172.21.15.84:0/2184344832 shutdown_connections 2024-03-29T11:27:18.172 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:18.177+0000 7f1df904a640 1 -- 172.21.15.84:0/2184344832 >> 172.21.15.84:0/2184344832 conn(0x7f1df40b35e0 msgr2=0x7f1df40b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:27:18.172 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:18.178+0000 7f1df904a640 1 -- 172.21.15.84:0/2184344832 shutdown_connections 2024-03-29T11:27:18.172 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:18.180+0000 7f1df904a640 1 -- 172.21.15.84:0/2184344832 wait complete. 2024-03-29T11:27:18.172 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:18.181+0000 7f1df904a640 10 monclient: build_initial_monmap 2024-03-29T11:27:18.172 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:18.181+0000 7f1df904a640 10 monclient: monmap: 2024-03-29T11:27:18.172 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-29T11:27:18.172 INFO:teuthology.misc.health.smithi084.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:27:18.172 INFO:teuthology.misc.health.smithi084.stderr:last_changed 0.000000 2024-03-29T11:27:18.172 INFO:teuthology.misc.health.smithi084.stderr:created 0.000000 2024-03-29T11:27:18.173 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-29T11:27:18.173 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:27:18.173 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-29T11:27:18.173 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.noname-b 2024-03-29T11:27:18.173 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-c 2024-03-29T11:27:18.173 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:27:18.173 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:18.181+0000 7f1df904a640 1 librados: starting msgr at 2024-03-29T11:27:18.173 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:18.181+0000 7f1df904a640 1 librados: starting objecter 2024-03-29T11:27:18.173 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:18.182+0000 7f1df904a640 1 Processor -- start 2024-03-29T11:27:18.173 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:18.182+0000 7f1df904a640 1 -- start start 2024-03-29T11:27:18.173 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:18.182+0000 7f1df904a640 1 librados: setting wanted keys 2024-03-29T11:27:18.173 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:18.182+0000 7f1df904a640 1 librados: calling monclient init 2024-03-29T11:27:18.173 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:18.182+0000 7f1df904a640 10 monclient: init 2024-03-29T11:27:18.174 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:18.183+0000 7f1df904a640 10 monclient: _reopen_session rank -1 2024-03-29T11:27:18.174 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:18.183+0000 7f1df904a640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:27:18.174 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:18.183+0000 7f1df904a640 10 monclient(hunting): picked mon.noname-c con 0x7f1df40a9ac0 addr v1:172.21.15.84:6790/0 2024-03-29T11:27:18.174 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:18.183+0000 7f1df904a640 10 monclient(hunting): picked mon.noname-b con 0x7f1df40af900 addr v1:172.21.15.179:6789/0 2024-03-29T11:27:18.174 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:18.183+0000 7f1df904a640 10 monclient(hunting): picked mon.noname-a con 0x7f1df413ee90 addr v1:172.21.15.84:6789/0 2024-03-29T11:27:18.174 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:18.183+0000 7f1df904a640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1df41e8050 con 0x7f1df413ee90 2024-03-29T11:27:18.174 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:18.183+0000 7f1df904a640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1df41c6460 con 0x7f1df40af900 2024-03-29T11:27:18.174 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:18.183+0000 7f1df904a640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1df41e7ce0 con 0x7f1df40a9ac0 2024-03-29T11:27:18.174 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:18.183+0000 7f1df904a640 10 monclient(hunting): _renew_subs 2024-03-29T11:27:18.174 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:18.183+0000 7f1df904a640 10 monclient(hunting): authenticate will time out at 1299.478607s 2024-03-29T11:27:18.175 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:18.186+0000 7f1df259c640 1 --1- >> v1:172.21.15.179:6789/0 conn(0x7f1df40af900 0x7f1df41c4460 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.179:6789/0 says I am v1:172.21.15.84:49436/0 (socket says 172.21.15.84:49436) 2024-03-29T11:27:18.175 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:18.186+0000 7f1df259c640 1 -- 172.21.15.84:0/2747014862 learned_addr learned my addr 172.21.15.84:0/2747014862 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-29T11:27:18.180 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:18.187+0000 7f1ddb7fe640 10 client.?.objecter ms_handle_connect 0x7f1df40a9ac0 2024-03-29T11:27:18.180 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:18.187+0000 7f1ddb7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:27:18.180 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:18.187+0000 7f1ddb7fe640 10 client.?.objecter ms_handle_connect 0x7f1df40af900 2024-03-29T11:27:18.180 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:18.187+0000 7f1ddb7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:27:18.180 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:18.187+0000 7f1ddb7fe640 1 -- 172.21.15.84:0/2747014862 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3351112353 0 0) 0x7f1df41e7ce0 con 0x7f1df40a9ac0 2024-03-29T11:27:18.180 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:18.187+0000 7f1ddb7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:27:18.180 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:18.187+0000 7f1ddb7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:27:18.180 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:18.187+0000 7f1ddb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:27:18.180 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:18.187+0000 7f1ddb7fe640 10 monclient(hunting): my global_id is 4538 2024-03-29T11:27:18.180 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:18.187+0000 7f1ddb7fe640 1 -- 172.21.15.84:0/2747014862 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1de0004eb0 con 0x7f1df40a9ac0 2024-03-29T11:27:18.181 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:18.187+0000 7f1ddb7fe640 1 -- 172.21.15.84:0/2747014862 <== mon.2 v1:172.21.15.84:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2307631098 0 0) 0x7f1de0004eb0 con 0x7f1df40a9ac0 2024-03-29T11:27:18.181 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:18.188+0000 7f1ddb7fe640 1 -- 172.21.15.84:0/2747014862 --> v1:172.21.15.84:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1df41e7ce0 con 0x7f1df40a9ac0 2024-03-29T11:27:18.181 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:18.188+0000 7f1ddb7fe640 10 client.?.objecter ms_handle_connect 0x7f1df413ee90 2024-03-29T11:27:18.181 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:18.188+0000 7f1ddb7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:27:18.181 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:18.188+0000 7f1ddb7fe640 1 -- 172.21.15.84:0/2747014862 <== mon.2 v1:172.21.15.84:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4236868694 0 0) 0x7f1de4001bc0 con 0x7f1df40a9ac0 2024-03-29T11:27:18.181 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:18.188+0000 7f1ddb7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:27:18.181 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:18.188+0000 7f1ddb7fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:27:18.181 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:18.188+0000 7f1ddb7fe640 10 monclient(hunting): dump: 2024-03-29T11:27:18.181 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T11:27:18.181 INFO:teuthology.misc.health.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:27:18.181 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T11:18:53.864025+0000 2024-03-29T11:27:18.182 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T11:18:53.864025+0000 2024-03-29T11:27:18.182 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T11:27:18.182 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:27:18.182 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T11:27:18.182 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T11:27:18.182 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T11:27:18.182 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:27:18.182 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:18.188+0000 7f1ddb7fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:27:18.182 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:18.188+0000 7f1ddb7fe640 1 -- 172.21.15.84:0/2747014862 <== mon.2 v1:172.21.15.84:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2257250039 0 0) 0x7f1df41e7ce0 con 0x7f1df40a9ac0 2024-03-29T11:27:18.182 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:18.188+0000 7f1ddb7fe640 1 -- 172.21.15.84:0/2747014862 >> v1:172.21.15.179:6789/0 conn(0x7f1df40af900 legacy=0x7f1df41c4460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:18.182 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:18.188+0000 7f1ddb7fe640 1 -- 172.21.15.84:0/2747014862 >> v1:172.21.15.84:6789/0 conn(0x7f1df413ee90 legacy=0x7f1df41c4b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:18.182 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:18.188+0000 7f1ddb7fe640 10 monclient: _finish_hunting 0 2024-03-29T11:27:18.183 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:18.188+0000 7f1ddb7fe640 1 monclient: found mon.c 2024-03-29T11:27:18.183 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:18.188+0000 7f1ddb7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:27:18.183 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:18.188+0000 7f1ddb7fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-29T11:27:18.183 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:18.188+0000 7f1ddb7fe640 1 -- 172.21.15.84:0/2747014862 --> v1:172.21.15.84:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1df4072570 con 0x7f1df40a9ac0 2024-03-29T11:27:18.183 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:18.188+0000 7f1ddb7fe640 10 monclient: _finish_auth 0 2024-03-29T11:27:18.183 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:18.188+0000 7f1ddb7fe640 10 monclient: _check_auth_tickets 2024-03-29T11:27:18.183 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:18.188+0000 7f1ddb7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:27:18.183 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:18.188+0000 7f1df904a640 5 monclient: authenticate success, global_id 4538 2024-03-29T11:27:18.183 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:18.188+0000 7f1df904a640 10 monclient: _renew_subs 2024-03-29T11:27:18.183 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:18.188+0000 7f1df904a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-29T11:27:18.183 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:18.188+0000 7f1df904a640 1 -- 172.21.15.84:0/2747014862 --> v1:172.21.15.84:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1df41c8490 con 0x7f1df40a9ac0 2024-03-29T11:27:18.183 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:18.188+0000 7f1df904a640 10 client.4538.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:27:18.183 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:18.188+0000 7f1df904a640 10 monclient: _renew_subs 2024-03-29T11:27:18.183 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:18.188+0000 7f1df904a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-29T11:27:18.183 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:18.188+0000 7f1df904a640 1 -- 172.21.15.84:0/2747014862 --> v1:172.21.15.84:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f1df41efa90 con 0x7f1df40a9ac0 2024-03-29T11:27:18.184 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:18.188+0000 7f1df904a640 1 librados: init done 2024-03-29T11:27:18.184 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:18.189+0000 7f1ddb7fe640 1 -- 172.21.15.84:0/2747014862 <== mon.2 v1:172.21.15.84:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1de4004a90 con 0x7f1df40a9ac0 2024-03-29T11:27:18.184 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:18.189+0000 7f1ddb7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:27:18.184 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:18.189+0000 7f1ddb7fe640 1 -- 172.21.15.84:0/2747014862 <== mon.2 v1:172.21.15.84:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4236868694 0 0) 0x7f1de4001250 con 0x7f1df40a9ac0 2024-03-29T11:27:18.184 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:18.189+0000 7f1ddb7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:27:18.184 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:18.189+0000 7f1ddb7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:27:18.184 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:18.189+0000 7f1ddb7fe640 10 monclient: dump: 2024-03-29T11:27:18.184 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T11:27:18.184 INFO:teuthology.misc.health.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:27:18.184 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T11:18:53.864025+0000 2024-03-29T11:27:18.184 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T11:18:53.864025+0000 2024-03-29T11:27:18.184 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T11:27:18.184 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:27:18.184 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T11:27:18.184 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T11:27:18.185 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T11:27:18.185 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:27:18.185 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:18.189+0000 7f1df0d99640 4 set_mon_vals no callback set 2024-03-29T11:27:18.185 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:18.191+0000 7f1df904a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:27:18.185 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:18.191+0000 7f1df904a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:27:18.185 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:18.191+0000 7f1df904a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-29T11:27:18.185 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:18.191+0000 7f1df904a640 1 -- 172.21.15.84:0/2747014862 --> v1:172.21.15.84:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1dd4005080 con 0x7f1df40a9ac0 2024-03-29T11:27:18.185 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:18.191+0000 7f1ddb7fe640 1 -- 172.21.15.84:0/2747014862 <== mon.2 v1:172.21.15.84:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4033942909 0 0) 0x7f1de4004ee0 con 0x7f1df40a9ac0 2024-03-29T11:27:18.186 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:18.192+0000 7f1ddb7fe640 1 -- 172.21.15.84:0/2747014862 <== mon.2 v1:172.21.15.84:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2808139587 0 0) 0x7f1de4084ba0 con 0x7f1df40a9ac0 2024-03-29T11:27:18.186 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:18.192+0000 7f1ddb7fe640 10 client.4538.objecter ms_dispatch 0x7f1df4000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:27:18.186 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:18.192+0000 7f1ddb7fe640 3 client.4538.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:27:18.186 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:18.192+0000 7f1ddb7fe640 3 client.4538.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:27:18.186 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:18.192+0000 7f1df259c640 10 monclient: get_auth_request con 0x7f1de006dd20 auth_method 0 2024-03-29T11:27:18.186 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:18.192+0000 7f1ddb7fe640 20 client.4538.objecter dump_active .. 0 homeless 2024-03-29T11:27:18.186 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:18.192+0000 7f1ddb7fe640 10 client.4538.objecter ms_handle_connect 0x7f1de006dd20 2024-03-29T11:27:18.189 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:18.200+0000 7f1ddb7fe640 1 -- 172.21.15.84:0/2747014862 <== mon.2 v1:172.21.15.84:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f1de4050480 con 0x7f1df40a9ac0 2024-03-29T11:27:18.189 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:18.200+0000 7f1ddb7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:27:18.189 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:18.200+0000 7f1ddb7fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:27:18.517 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:18.528+0000 7f1df904a640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:27:18.517 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:18.528+0000 7f1df904a640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:27:18.517 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:18.528+0000 7f1df904a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-29T11:27:18.517 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:18.528+0000 7f1df904a640 1 -- 172.21.15.84:0/2747014862 --> v1:172.21.15.84:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1dd4005330 con 0x7f1df40a9ac0 2024-03-29T11:27:18.518 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:18.529+0000 7f1ddb7fe640 1 -- 172.21.15.84:0/2747014862 <== mon.2 v1:172.21.15.84:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4209477549) 0x7f1de40523b0 con 0x7f1df40a9ac0 2024-03-29T11:27:18.518 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:18.529+0000 7f1ddb7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:27:18.518 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:18.529+0000 7f1ddb7fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:27:18.518 INFO:teuthology.misc.health.smithi084.stdout: 2024-03-29T11:27:18.519 INFO:teuthology.misc.health.smithi084.stdout:{"epoch":10,"fsid":"2c3e002f-9eb0-4860-853b-4ef1ef562213","created":"2024-03-29T11:19:32.029169+0000","modified":"2024-03-29T11:19:34.200299+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:34.200299+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"5c39e276-2034-4716-bbc0-57a634eefa92","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"cd580e99-109f-4bf1-88c0-a5987bc8b9ed","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"11e60a5a-051d-453f-a190-66387cb35592","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"2c329fbe-bbc4-4030-bf87-d711cad04679","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f4213d4a-bd36-4433-91ae-d7e2b02481d3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"f0cf44ea-6055-48af-805b-aed858102ff9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"d895eaa0-cc0b-4ee4-9b5a-2dfcd89a60b7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"9bc5a7bc-8f52-4db9-96d6-de5e68d52eaa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:27:18.522 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:18.532+0000 7f1df904a640 10 librados: watch_flush enter 2024-03-29T11:27:18.522 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:18.532+0000 7f1df904a640 10 librados: watch_flush exit 2024-03-29T11:27:18.522 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:18.532+0000 7f1df904a640 20 client.4538.objecter shutdown clearing up homeless session... 2024-03-29T11:27:18.522 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:18.532+0000 7f1df904a640 10 client.4538.objecter successfully canceled tick 2024-03-29T11:27:18.522 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:18.533+0000 7f1df904a640 1 -- 172.21.15.84:0/2747014862 >> v1:172.21.15.179:6816/2981134980 conn(0x7f1de006dd20 legacy=0x7f1de00701e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:18.522 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:18.533+0000 7f1df904a640 10 monclient: shutdown 2024-03-29T11:27:18.522 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:18.533+0000 7f1df904a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:27:18.522 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:18.533+0000 7f1df904a640 1 -- 172.21.15.84:0/2747014862 >> v1:172.21.15.84:6790/0 conn(0x7f1df40a9ac0 legacy=0x7f1df41e75d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:18.522 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:18.533+0000 7f1df904a640 1 -- 172.21.15.84:0/2747014862 shutdown_connections 2024-03-29T11:27:18.522 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:18.533+0000 7f1df904a640 1 -- 172.21.15.84:0/2747014862 >> 172.21.15.84:0/2747014862 conn(0x7f1df40b35e0 msgr2=0x7f1df40ad280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:27:18.523 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:18.533+0000 7f1df904a640 1 -- 172.21.15.84:0/2747014862 shutdown_connections 2024-03-29T11:27:18.523 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:18.533+0000 7f1df904a640 1 -- 172.21.15.84:0/2747014862 wait complete. 2024-03-29T11:27:18.523 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:18.533+0000 7f1df904a640 1 librados: shutdown 2024-03-29T11:27:18.539 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:27:25.346 DEBUG:teuthology.orchestra.run.smithi084:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:27:25.482 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:25.493+0000 7f8db759e640 10 monclient: get_monmap_and_config 2024-03-29T11:27:25.482 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:25.493+0000 7f8db759e640 10 monclient: build_initial_monmap 2024-03-29T11:27:25.482 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:25.493+0000 7f8db759e640 10 monclient: monmap: 2024-03-29T11:27:25.482 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-29T11:27:25.482 INFO:teuthology.misc.health.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:27:25.482 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T11:27:25.494770+0000 2024-03-29T11:27:25.482 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T11:27:25.494770+0000 2024-03-29T11:27:25.482 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-29T11:27:25.483 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:27:25.483 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-29T11:27:25.483 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.noname-c 2024-03-29T11:27:25.483 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-b 2024-03-29T11:27:25.483 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:27:25.484 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:25.494+0000 7f8db759e640 1 Processor -- start 2024-03-29T11:27:25.484 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:25.495+0000 7f8db759e640 1 -- start start 2024-03-29T11:27:25.484 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:25.495+0000 7f8db759e640 10 monclient: init 2024-03-29T11:27:25.484 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:25.496+0000 7f8db759e640 10 monclient: _reopen_session rank -1 2024-03-29T11:27:25.484 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:25.496+0000 7f8db759e640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:27:25.485 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:25.496+0000 7f8db759e640 10 monclient(hunting): picked mon.noname-a con 0x7f8db00389e0 addr v1:172.21.15.84:6789/0 2024-03-29T11:27:25.485 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:25.496+0000 7f8db759e640 10 monclient(hunting): picked mon.noname-c con 0x7f8db0038dc0 addr v1:172.21.15.179:6789/0 2024-03-29T11:27:25.485 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:25.496+0000 7f8db759e640 10 monclient(hunting): picked mon.noname-b con 0x7f8db0030660 addr v1:172.21.15.84:6790/0 2024-03-29T11:27:25.485 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:25.496+0000 7f8db759e640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8db00c54c0 con 0x7f8db00389e0 2024-03-29T11:27:25.485 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:25.496+0000 7f8db759e640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8db0036430 con 0x7f8db0038dc0 2024-03-29T11:27:25.485 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:25.496+0000 7f8db6d9d640 1 --1- >> v1:172.21.15.84:6790/0 conn(0x7f8db0030660 0x7f8db00d3170 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.84:6790/0 says I am v1:172.21.15.84:58058/0 (socket says 172.21.15.84:58058) 2024-03-29T11:27:25.485 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:25.496+0000 7f8db6d9d640 1 -- 172.21.15.84:0/3243009450 learned_addr learned my addr 172.21.15.84:0/3243009450 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-29T11:27:25.485 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:25.497+0000 7f8db759e640 1 -- 172.21.15.84:0/3243009450 --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8db0039840 con 0x7f8db0030660 2024-03-29T11:27:25.486 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:25.497+0000 7f8db759e640 10 monclient(hunting): _renew_subs 2024-03-29T11:27:25.486 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:25.497+0000 7f8db559a640 1 -- 172.21.15.84:0/3243009450 <== mon.1 v1:172.21.15.179:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2544906206 0 0) 0x7f8db0036430 con 0x7f8db0038dc0 2024-03-29T11:27:25.486 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:25.497+0000 7f8db759e640 10 monclient(hunting): authenticate will time out at 1306.791299s 2024-03-29T11:27:25.486 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:25.497+0000 7f8db559a640 10 monclient(hunting): _negotiate 2024-03-29T11:27:25.486 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:25.497+0000 7f8db559a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:27:25.486 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:25.497+0000 7f8db559a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:27:25.486 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:25.497+0000 7f8db559a640 10 monclient(hunting): my global_id is 4462 2024-03-29T11:27:25.486 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:25.497+0000 7f8db559a640 1 -- 172.21.15.84:0/3243009450 --> v1:172.21.15.179:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8da0003520 con 0x7f8db0038dc0 2024-03-29T11:27:25.486 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:25.497+0000 7f8db559a640 1 -- 172.21.15.84:0/3243009450 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1148177170 0 0) 0x7f8db0039840 con 0x7f8db0030660 2024-03-29T11:27:25.486 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:25.497+0000 7f8db559a640 10 monclient(hunting): _negotiate 2024-03-29T11:27:25.487 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:25.497+0000 7f8db559a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:27:25.487 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:25.497+0000 7f8db559a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:27:25.487 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:25.497+0000 7f8db559a640 10 monclient(hunting): my global_id is 4541 2024-03-29T11:27:25.487 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:25.497+0000 7f8db559a640 1 -- 172.21.15.84:0/3243009450 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8db0036430 con 0x7f8db0030660 2024-03-29T11:27:25.487 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:25.498+0000 7f8db559a640 1 -- 172.21.15.84:0/3243009450 <== mon.2 v1:172.21.15.84:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3705885798 0 0) 0x7f8db0036430 con 0x7f8db0030660 2024-03-29T11:27:25.487 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:25.498+0000 7f8db559a640 1 -- 172.21.15.84:0/3243009450 >> v1:172.21.15.179:6789/0 conn(0x7f8db0038dc0 legacy=0x7f8db00cadd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:25.487 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:25.498+0000 7f8db559a640 1 -- 172.21.15.84:0/3243009450 >> v1:172.21.15.84:6789/0 conn(0x7f8db00389e0 legacy=0x7f8db0033520 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:27:25.491 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:25.499+0000 7f8db559a640 10 monclient: _finish_hunting 0 2024-03-29T11:27:25.491 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:25.499+0000 7f8db559a640 1 monclient: found mon.noname-b 2024-03-29T11:27:25.491 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:25.499+0000 7f8db559a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:27:25.491 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:25.499+0000 7f8db559a640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.84:6790/0 2024-03-29T11:27:25.491 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:25.499+0000 7f8db559a640 1 -- 172.21.15.84:0/3243009450 --> v1:172.21.15.84:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8db0035690 con 0x7f8db0030660 2024-03-29T11:27:25.491 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:25.499+0000 7f8db559a640 10 monclient: _finish_auth 0 2024-03-29T11:27:25.491 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:25.499+0000 7f8db559a640 10 monclient: _check_auth_tickets 2024-03-29T11:27:25.491 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:25.499+0000 7f8db559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:27:25.492 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:25.499+0000 7f8db559a640 1 -- 172.21.15.84:0/3243009450 <== mon.2 v1:172.21.15.84:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4236868694 0 0) 0x7f8db804f730 con 0x7f8db0030660 2024-03-29T11:27:25.492 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:25.499+0000 7f8db759e640 5 monclient: authenticate success, global_id 4541 2024-03-29T11:27:25.492 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:25.499+0000 7f8db759e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:27:25.492 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:25.499+0000 7f8db559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:27:25.492 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:25.499+0000 7f8db559a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:27:25.492 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:25.499+0000 7f8db559a640 10 monclient: dump: 2024-03-29T11:27:25.492 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T11:27:25.492 INFO:teuthology.misc.health.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:27:25.492 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T11:18:53.864025+0000 2024-03-29T11:27:25.492 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T11:18:53.864025+0000 2024-03-29T11:27:25.492 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T11:27:25.492 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:27:25.492 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T11:27:25.492 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T11:27:25.492 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T11:27:25.493 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:27:25.493 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:25.499+0000 7f8db759e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:27:25.493 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:25.499+0000 7f8db559a640 1 -- 172.21.15.84:0/3243009450 <== mon.2 v1:172.21.15.84:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8db8075f10 con 0x7f8db0030660 2024-03-29T11:27:25.493 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:25.499+0000 7f8db559a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:27:25.493 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:25.499+0000 7f8db559a640 1 -- 172.21.15.84:0/3243009450 <== mon.2 v1:172.21.15.84:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4236868694 0 0) 0x7f8db80758b0 con 0x7f8db0030660 2024-03-29T11:27:25.493 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:25.499+0000 7f8db559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:27:25.493 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:25.499+0000 7f8db559a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:27:25.493 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:25.499+0000 7f8db559a640 10 monclient: dump: 2024-03-29T11:27:25.493 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T11:27:25.493 INFO:teuthology.misc.health.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:27:25.493 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T11:18:53.864025+0000 2024-03-29T11:27:25.493 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T11:18:53.864025+0000 2024-03-29T11:27:25.493 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T11:27:25.493 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:27:25.493 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T11:27:25.494 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T11:27:25.494 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T11:27:25.494 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:27:25.494 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:25.499+0000 7f8db759e640 10 monclient: get_monmap_and_config success 2024-03-29T11:27:25.494 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:25.499+0000 7f8db759e640 4 set_mon_vals no callback set 2024-03-29T11:27:25.494 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:25.500+0000 7f8db759e640 10 monclient: shutdown 2024-03-29T11:27:25.494 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:25.500+0000 7f8db759e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:27:25.494 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:25.500+0000 7f8db759e640 1 -- 172.21.15.84:0/3243009450 >> v1:172.21.15.84:6790/0 conn(0x7f8db0030660 legacy=0x7f8db00d3170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:25.494 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:25.500+0000 7f8db759e640 1 -- 172.21.15.84:0/3243009450 shutdown_connections 2024-03-29T11:27:25.494 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:25.500+0000 7f8db759e640 1 -- 172.21.15.84:0/3243009450 >> 172.21.15.84:0/3243009450 conn(0x7f8db0005c00 msgr2=0x7f8db0006010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:27:25.495 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:25.500+0000 7f8db759e640 1 -- 172.21.15.84:0/3243009450 shutdown_connections 2024-03-29T11:27:25.495 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:25.501+0000 7f8db759e640 1 -- 172.21.15.84:0/3243009450 wait complete. 2024-03-29T11:27:25.495 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:25.502+0000 7f8db759e640 10 monclient: build_initial_monmap 2024-03-29T11:27:25.495 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:25.502+0000 7f8db759e640 10 monclient: monmap: 2024-03-29T11:27:25.495 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-29T11:27:25.495 INFO:teuthology.misc.health.smithi084.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:27:25.495 INFO:teuthology.misc.health.smithi084.stderr:last_changed 0.000000 2024-03-29T11:27:25.495 INFO:teuthology.misc.health.smithi084.stderr:created 0.000000 2024-03-29T11:27:25.496 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-29T11:27:25.496 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:27:25.496 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-29T11:27:25.496 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.noname-b 2024-03-29T11:27:25.496 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-c 2024-03-29T11:27:25.496 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:27:25.496 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:25.503+0000 7f8db759e640 1 librados: starting msgr at 2024-03-29T11:27:25.496 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:25.503+0000 7f8db759e640 1 librados: starting objecter 2024-03-29T11:27:25.496 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:25.503+0000 7f8db759e640 1 Processor -- start 2024-03-29T11:27:25.496 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:25.503+0000 7f8db759e640 1 -- start start 2024-03-29T11:27:25.496 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:25.503+0000 7f8db759e640 1 librados: setting wanted keys 2024-03-29T11:27:25.496 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:25.503+0000 7f8db759e640 1 librados: calling monclient init 2024-03-29T11:27:25.497 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:25.503+0000 7f8db759e640 10 monclient: init 2024-03-29T11:27:25.504 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:25.504+0000 7f8db759e640 10 monclient: _reopen_session rank -1 2024-03-29T11:27:25.504 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:25.504+0000 7f8db759e640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:27:25.504 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:25.504+0000 7f8db759e640 10 monclient(hunting): picked mon.noname-a con 0x7f8db0030660 addr v1:172.21.15.84:6789/0 2024-03-29T11:27:25.504 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:25.504+0000 7f8db759e640 10 monclient(hunting): picked mon.noname-c con 0x7f8db00389e0 addr v1:172.21.15.84:6790/0 2024-03-29T11:27:25.504 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:25.504+0000 7f8db759e640 10 monclient(hunting): picked mon.noname-b con 0x7f8db000ca70 addr v1:172.21.15.179:6789/0 2024-03-29T11:27:25.504 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:25.504+0000 7f8db759e640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8db000d8d0 con 0x7f8db0030660 2024-03-29T11:27:25.504 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:25.504+0000 7f8db759e640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8db00c69f0 con 0x7f8db000ca70 2024-03-29T11:27:25.504 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:25.504+0000 7f8db759e640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8db00c6cd0 con 0x7f8db00389e0 2024-03-29T11:27:25.504 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:25.504+0000 7f8db759e640 10 monclient(hunting): _renew_subs 2024-03-29T11:27:25.504 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:25.504+0000 7f8db759e640 10 monclient(hunting): authenticate will time out at 1306.798394s 2024-03-29T11:27:25.504 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:25.504+0000 7f8db5d9b640 1 --1- >> v1:172.21.15.84:6790/0 conn(0x7f8db00389e0 0x7f8db000a1f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.84:6790/0 says I am v1:172.21.15.84:40894/0 (socket says 172.21.15.84:40894) 2024-03-29T11:27:25.505 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:25.504+0000 7f8db5d9b640 1 -- 172.21.15.84:0/176032530 learned_addr learned my addr 172.21.15.84:0/176032530 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-29T11:27:25.505 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:25.505+0000 7f8da6ffd640 10 client.?.objecter ms_handle_connect 0x7f8db00389e0 2024-03-29T11:27:25.505 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:25.505+0000 7f8da6ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:27:25.505 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:25.505+0000 7f8da6ffd640 10 client.?.objecter ms_handle_connect 0x7f8db000ca70 2024-03-29T11:27:25.505 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:25.505+0000 7f8da6ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:27:25.505 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:25.505+0000 7f8da6ffd640 1 -- 172.21.15.84:0/176032530 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 65714099 0 0) 0x7f8db00c6cd0 con 0x7f8db00389e0 2024-03-29T11:27:25.505 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:25.505+0000 7f8da6ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:27:25.505 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:25.505+0000 7f8da6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:27:25.505 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:25.505+0000 7f8da6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:27:25.505 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:25.505+0000 7f8da6ffd640 10 monclient(hunting): my global_id is 4544 2024-03-29T11:27:25.505 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:25.505+0000 7f8da6ffd640 1 -- 172.21.15.84:0/176032530 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8d98004eb0 con 0x7f8db00389e0 2024-03-29T11:27:25.505 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:25.505+0000 7f8da6ffd640 1 -- 172.21.15.84:0/176032530 <== mon.1 v1:172.21.15.179:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2346132043 0 0) 0x7f8db00c69f0 con 0x7f8db000ca70 2024-03-29T11:27:25.505 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:25.505+0000 7f8da6ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:27:25.505 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:25.505+0000 7f8da6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:27:25.505 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:25.505+0000 7f8da6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:27:25.506 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:25.505+0000 7f8da6ffd640 10 monclient(hunting): my global_id is 4465 2024-03-29T11:27:25.506 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:25.505+0000 7f8da6ffd640 1 -- 172.21.15.84:0/176032530 --> v1:172.21.15.179:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8db00c6cd0 con 0x7f8db000ca70 2024-03-29T11:27:25.506 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:25.506+0000 7f8da6ffd640 1 -- 172.21.15.84:0/176032530 <== mon.2 v1:172.21.15.84:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 52615928 0 0) 0x7f8d98004eb0 con 0x7f8db00389e0 2024-03-29T11:27:25.506 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:25.506+0000 7f8da6ffd640 1 -- 172.21.15.84:0/176032530 --> v1:172.21.15.84:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8db00c69f0 con 0x7f8db00389e0 2024-03-29T11:27:25.506 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:25.506+0000 7f8da6ffd640 1 -- 172.21.15.84:0/176032530 <== mon.2 v1:172.21.15.84:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4236868694 0 0) 0x7f8dac001970 con 0x7f8db00389e0 2024-03-29T11:27:25.506 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:25.506+0000 7f8da6ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:27:25.506 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:25.506+0000 7f8da6ffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:27:25.506 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:25.506+0000 7f8da6ffd640 10 monclient(hunting): dump: 2024-03-29T11:27:25.506 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T11:27:25.506 INFO:teuthology.misc.health.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:27:25.506 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T11:18:53.864025+0000 2024-03-29T11:27:25.506 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T11:18:53.864025+0000 2024-03-29T11:27:25.506 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T11:27:25.507 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:27:25.507 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T11:27:25.507 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T11:27:25.507 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T11:27:25.507 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:27:25.507 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:25.506+0000 7f8da6ffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:27:25.507 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:25.506+0000 7f8da6ffd640 1 -- 172.21.15.84:0/176032530 <== mon.1 v1:172.21.15.179:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1735836964 0 0) 0x7f8db00c6cd0 con 0x7f8db000ca70 2024-03-29T11:27:25.507 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:25.506+0000 7f8da6ffd640 1 -- 172.21.15.84:0/176032530 --> v1:172.21.15.179:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8d98004eb0 con 0x7f8db000ca70 2024-03-29T11:27:25.507 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:25.506+0000 7f8da6ffd640 1 -- 172.21.15.84:0/176032530 <== mon.1 v1:172.21.15.179:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4236868694 0 0) 0x7f8db804f400 con 0x7f8db000ca70 2024-03-29T11:27:25.507 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:25.506+0000 7f8da6ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:27:25.507 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:25.506+0000 7f8da6ffd640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:27:25.507 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:25.506+0000 7f8da6ffd640 10 monclient(hunting): dump: 2024-03-29T11:27:25.507 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T11:27:25.507 INFO:teuthology.misc.health.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:27:25.508 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T11:18:53.864025+0000 2024-03-29T11:27:25.508 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T11:18:53.864025+0000 2024-03-29T11:27:25.508 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T11:27:25.508 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:27:25.508 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T11:27:25.508 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T11:27:25.508 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T11:27:25.508 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:27:25.508 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:25.506+0000 7f8da6ffd640 1 -- 172.21.15.84:0/176032530 <== mon.2 v1:172.21.15.84:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 932084938 0 0) 0x7f8db00c69f0 con 0x7f8db00389e0 2024-03-29T11:27:25.509 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:25.506+0000 7f8da6ffd640 1 -- 172.21.15.84:0/176032530 >> v1:172.21.15.179:6789/0 conn(0x7f8db000ca70 legacy=0x7f8db016b480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:25.509 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:25.506+0000 7f8da6ffd640 1 -- 172.21.15.84:0/176032530 >> v1:172.21.15.84:6789/0 conn(0x7f8db0030660 legacy=0x7f8db0009ae0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:27:25.509 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:25.507+0000 7f8da6ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:27:25.509 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:25.507+0000 7f8da6ffd640 1 monclient: found mon.c 2024-03-29T11:27:25.509 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:25.507+0000 7f8da6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:27:25.509 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:25.507+0000 7f8da6ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-29T11:27:25.509 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:25.507+0000 7f8da6ffd640 1 -- 172.21.15.84:0/176032530 --> v1:172.21.15.84:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8db000bd80 con 0x7f8db00389e0 2024-03-29T11:27:25.509 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:25.507+0000 7f8da6ffd640 10 monclient: _finish_auth 0 2024-03-29T11:27:25.509 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:25.507+0000 7f8da6ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:27:25.510 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:25.507+0000 7f8da6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:27:25.510 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:25.507+0000 7f8db759e640 5 monclient: authenticate success, global_id 4544 2024-03-29T11:27:25.510 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:25.507+0000 7f8db759e640 10 monclient: _renew_subs 2024-03-29T11:27:25.510 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:25.507+0000 7f8db759e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-29T11:27:25.510 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:25.507+0000 7f8db759e640 1 -- 172.21.15.84:0/176032530 --> v1:172.21.15.84:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8db00c29b0 con 0x7f8db00389e0 2024-03-29T11:27:25.510 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:25.507+0000 7f8db759e640 10 client.4544.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:27:25.510 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:25.507+0000 7f8db759e640 10 monclient: _renew_subs 2024-03-29T11:27:25.510 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:25.507+0000 7f8db759e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-29T11:27:25.510 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:25.507+0000 7f8db759e640 1 -- 172.21.15.84:0/176032530 --> v1:172.21.15.84:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f8db016cd80 con 0x7f8db00389e0 2024-03-29T11:27:25.510 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:25.507+0000 7f8db759e640 1 librados: init done 2024-03-29T11:27:25.511 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:25.507+0000 7f8da6ffd640 1 -- 172.21.15.84:0/176032530 <== mon.2 v1:172.21.15.84:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8dac001c70 con 0x7f8db00389e0 2024-03-29T11:27:25.511 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:25.507+0000 7f8da6ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:27:25.511 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:25.507+0000 7f8da6ffd640 1 -- 172.21.15.84:0/176032530 <== mon.2 v1:172.21.15.84:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4236868694 0 0) 0x7f8dac003a60 con 0x7f8db00389e0 2024-03-29T11:27:25.511 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:25.507+0000 7f8da6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:27:25.511 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:25.507+0000 7f8da6ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:27:25.511 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:25.507+0000 7f8da6ffd640 10 monclient: dump: 2024-03-29T11:27:25.511 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T11:27:25.511 INFO:teuthology.misc.health.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:27:25.511 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T11:18:53.864025+0000 2024-03-29T11:27:25.511 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T11:18:53.864025+0000 2024-03-29T11:27:25.511 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T11:27:25.511 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:27:25.511 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T11:27:25.511 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T11:27:25.512 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T11:27:25.512 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:27:25.512 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:25.509+0000 7f8db4d99640 4 set_mon_vals no callback set 2024-03-29T11:27:25.512 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:25.509+0000 7f8db759e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:27:25.512 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:25.510+0000 7f8da6ffd640 1 -- 172.21.15.84:0/176032530 <== mon.2 v1:172.21.15.84:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4033942909 0 0) 0x7f8dac003bc0 con 0x7f8db00389e0 2024-03-29T11:27:25.512 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:25.510+0000 7f8db759e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:27:25.512 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:25.510+0000 7f8db759e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-29T11:27:25.512 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:25.510+0000 7f8db759e640 1 -- 172.21.15.84:0/176032530 --> v1:172.21.15.84:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8db0007b70 con 0x7f8db00389e0 2024-03-29T11:27:25.512 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:25.510+0000 7f8da6ffd640 1 -- 172.21.15.84:0/176032530 <== mon.2 v1:172.21.15.84:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2808139587 0 0) 0x7f8dac084aa0 con 0x7f8db00389e0 2024-03-29T11:27:25.512 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:25.510+0000 7f8da6ffd640 10 client.4544.objecter ms_dispatch 0x7f8db8000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:27:25.512 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:25.511+0000 7f8da6ffd640 3 client.4544.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:27:25.512 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:25.511+0000 7f8da6ffd640 3 client.4544.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:27:25.512 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:25.511+0000 7f8da6ffd640 20 client.4544.objecter dump_active .. 0 homeless 2024-03-29T11:27:25.512 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:25.511+0000 7f8db659c640 10 monclient: get_auth_request con 0x7f8d9806dd70 auth_method 0 2024-03-29T11:27:25.512 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:25.512+0000 7f8da6ffd640 10 client.4544.objecter ms_handle_connect 0x7f8d9806dd70 2024-03-29T11:27:25.513 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:25.519+0000 7f8da6ffd640 1 -- 172.21.15.84:0/176032530 <== mon.2 v1:172.21.15.84:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f8dac050380 con 0x7f8db00389e0 2024-03-29T11:27:25.513 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:25.519+0000 7f8da6ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:27:25.513 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:25.519+0000 7f8da6ffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:27:25.830 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:25.842+0000 7f8db759e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:27:25.830 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:25.842+0000 7f8db759e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:27:25.830 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:25.842+0000 7f8db759e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-29T11:27:25.830 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:25.842+0000 7f8db759e640 1 -- 172.21.15.84:0/176032530 --> v1:172.21.15.84:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8db0009800 con 0x7f8db00389e0 2024-03-29T11:27:25.831 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:25.843+0000 7f8da6ffd640 1 -- 172.21.15.84:0/176032530 <== mon.2 v1:172.21.15.84:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4209477549) 0x7f8db0009800 con 0x7f8db00389e0 2024-03-29T11:27:25.831 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:25.843+0000 7f8da6ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:27:25.831 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:25.843+0000 7f8da6ffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:27:25.832 INFO:teuthology.misc.health.smithi084.stdout: 2024-03-29T11:27:25.833 INFO:teuthology.misc.health.smithi084.stdout:{"epoch":10,"fsid":"2c3e002f-9eb0-4860-853b-4ef1ef562213","created":"2024-03-29T11:19:32.029169+0000","modified":"2024-03-29T11:19:34.200299+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:34.200299+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"5c39e276-2034-4716-bbc0-57a634eefa92","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"cd580e99-109f-4bf1-88c0-a5987bc8b9ed","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"11e60a5a-051d-453f-a190-66387cb35592","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"2c329fbe-bbc4-4030-bf87-d711cad04679","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f4213d4a-bd36-4433-91ae-d7e2b02481d3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"f0cf44ea-6055-48af-805b-aed858102ff9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"d895eaa0-cc0b-4ee4-9b5a-2dfcd89a60b7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"9bc5a7bc-8f52-4db9-96d6-de5e68d52eaa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:27:25.836 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:25.848+0000 7f8db759e640 10 librados: watch_flush enter 2024-03-29T11:27:25.836 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:25.848+0000 7f8db759e640 10 librados: watch_flush exit 2024-03-29T11:27:25.836 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:25.848+0000 7f8db759e640 20 client.4544.objecter shutdown clearing up homeless session... 2024-03-29T11:27:25.836 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:25.848+0000 7f8db759e640 10 client.4544.objecter successfully canceled tick 2024-03-29T11:27:25.837 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:25.848+0000 7f8db759e640 1 -- 172.21.15.84:0/176032530 >> v1:172.21.15.179:6816/2981134980 conn(0x7f8d9806dd70 legacy=0x7f8d98070230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:25.837 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:25.848+0000 7f8db759e640 10 monclient: shutdown 2024-03-29T11:27:25.837 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:25.848+0000 7f8db759e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:27:25.837 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:25.848+0000 7f8db759e640 1 -- 172.21.15.84:0/176032530 >> v1:172.21.15.84:6790/0 conn(0x7f8db00389e0 legacy=0x7f8db000a1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:25.837 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:25.849+0000 7f8db759e640 1 -- 172.21.15.84:0/176032530 shutdown_connections 2024-03-29T11:27:25.838 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:25.849+0000 7f8db759e640 1 -- 172.21.15.84:0/176032530 >> 172.21.15.84:0/176032530 conn(0x7f8db0005c00 msgr2=0x7f8db0034090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:27:25.838 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:25.849+0000 7f8db759e640 1 -- 172.21.15.84:0/176032530 shutdown_connections 2024-03-29T11:27:25.838 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:25.849+0000 7f8db759e640 1 -- 172.21.15.84:0/176032530 wait complete. 2024-03-29T11:27:25.838 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:25.849+0000 7f8db759e640 1 librados: shutdown 2024-03-29T11:27:25.854 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:27:32.662 DEBUG:teuthology.orchestra.run.smithi084:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:27:32.810 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:32.821+0000 7f0704a4e640 10 monclient: get_monmap_and_config 2024-03-29T11:27:32.810 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:32.821+0000 7f0704a4e640 10 monclient: build_initial_monmap 2024-03-29T11:27:32.810 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:32.821+0000 7f0704a4e640 10 monclient: monmap: 2024-03-29T11:27:32.810 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-29T11:27:32.810 INFO:teuthology.misc.health.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:27:32.810 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T11:27:32.822681+0000 2024-03-29T11:27:32.810 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T11:27:32.822681+0000 2024-03-29T11:27:32.810 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-29T11:27:32.810 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:27:32.810 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-29T11:27:32.811 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.noname-c 2024-03-29T11:27:32.811 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-b 2024-03-29T11:27:32.811 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:27:32.811 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:32.822+0000 7f0704a4e640 1 Processor -- start 2024-03-29T11:27:32.811 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:32.823+0000 7f0704a4e640 1 -- start start 2024-03-29T11:27:32.811 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:32.823+0000 7f0704a4e640 10 monclient: init 2024-03-29T11:27:32.812 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:32.824+0000 7f0704a4e640 10 monclient: _reopen_session rank -1 2024-03-29T11:27:32.812 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:32.824+0000 7f0704a4e640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:27:32.812 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:32.824+0000 7f0704a4e640 10 monclient(hunting): picked mon.noname-a con 0x7f07001457f0 addr v1:172.21.15.84:6789/0 2024-03-29T11:27:32.812 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:32.824+0000 7f0704a4e640 10 monclient(hunting): picked mon.noname-b con 0x7f070013f650 addr v1:172.21.15.84:6790/0 2024-03-29T11:27:32.815 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:32.824+0000 7f0704a4e640 10 monclient(hunting): picked mon.noname-c con 0x7f07000a9d70 addr v1:172.21.15.179:6789/0 2024-03-29T11:27:32.815 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:32.826+0000 7f06fdd9b640 1 --1- >> v1:172.21.15.84:6790/0 conn(0x7f070013f650 0x7f07000b2620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.84:6790/0 says I am v1:172.21.15.84:49966/0 (socket says 172.21.15.84:49966) 2024-03-29T11:27:32.815 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:32.826+0000 7f06fdd9b640 1 -- 172.21.15.84:0/1985635217 learned_addr learned my addr 172.21.15.84:0/1985635217 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-29T11:27:32.815 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:32.826+0000 7f0704a4e640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f07000aaef0 con 0x7f07001457f0 2024-03-29T11:27:32.815 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:32.827+0000 7f0704a4e640 1 -- 172.21.15.84:0/1985635217 --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0700145bd0 con 0x7f07000a9d70 2024-03-29T11:27:32.815 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:32.827+0000 7f0704a4e640 1 -- 172.21.15.84:0/1985635217 --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0700145eb0 con 0x7f070013f650 2024-03-29T11:27:32.815 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:32.827+0000 7f0704a4e640 10 monclient(hunting): _renew_subs 2024-03-29T11:27:32.815 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:32.827+0000 7f0704a4e640 10 monclient(hunting): authenticate will time out at 1314.121374s 2024-03-29T11:27:32.816 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:32.827+0000 7f06fd59a640 1 -- 172.21.15.84:0/1985635217 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2224250755 0 0) 0x7f0700145eb0 con 0x7f070013f650 2024-03-29T11:27:32.816 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:32.827+0000 7f06fd59a640 10 monclient(hunting): _negotiate 2024-03-29T11:27:32.816 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:32.827+0000 7f06fd59a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:27:32.816 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:32.827+0000 7f06fd59a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:27:32.817 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:32.828+0000 7f06fd59a640 10 monclient(hunting): my global_id is 4547 2024-03-29T11:27:32.817 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:32.828+0000 7f06fd59a640 1 -- 172.21.15.84:0/1985635217 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f06e8003520 con 0x7f070013f650 2024-03-29T11:27:32.817 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:32.828+0000 7f06fd59a640 1 -- 172.21.15.84:0/1985635217 <== mon.2 v1:172.21.15.84:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 120416128 0 0) 0x7f06e8003520 con 0x7f070013f650 2024-03-29T11:27:32.817 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:32.828+0000 7f06fd59a640 1 -- 172.21.15.84:0/1985635217 >> v1:172.21.15.179:6789/0 conn(0x7f07000a9d70 legacy=0x7f070014d8c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:27:32.818 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:32.829+0000 7f06fd59a640 1 -- 172.21.15.84:0/1985635217 >> v1:172.21.15.84:6789/0 conn(0x7f07001457f0 legacy=0x7f070013ef40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:27:32.819 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:32.829+0000 7f06fd59a640 10 monclient: _finish_hunting 0 2024-03-29T11:27:32.819 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:32.830+0000 7f06fd59a640 1 monclient: found mon.noname-b 2024-03-29T11:27:32.819 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:32.830+0000 7f06fd59a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:27:32.819 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:32.830+0000 7f06fd59a640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.84:6790/0 2024-03-29T11:27:32.819 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:32.830+0000 7f06fd59a640 1 -- 172.21.15.84:0/1985635217 --> v1:172.21.15.84:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0700146340 con 0x7f070013f650 2024-03-29T11:27:32.819 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:32.830+0000 7f06fd59a640 10 monclient: _finish_auth 0 2024-03-29T11:27:32.819 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:32.830+0000 7f06fd59a640 10 monclient: _check_auth_tickets 2024-03-29T11:27:32.819 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:32.830+0000 7f06fd59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:27:32.819 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:32.830+0000 7f06fd59a640 1 -- 172.21.15.84:0/1985635217 <== mon.2 v1:172.21.15.84:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4236868694 0 0) 0x7f06f0001690 con 0x7f070013f650 2024-03-29T11:27:32.819 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:32.830+0000 7f06fd59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:27:32.819 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:32.830+0000 7f06fd59a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:27:32.819 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:32.830+0000 7f06fd59a640 10 monclient: dump: 2024-03-29T11:27:32.819 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T11:27:32.819 INFO:teuthology.misc.health.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:27:32.819 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T11:18:53.864025+0000 2024-03-29T11:27:32.820 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T11:18:53.864025+0000 2024-03-29T11:27:32.820 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T11:27:32.820 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:27:32.820 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T11:27:32.820 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T11:27:32.820 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T11:27:32.820 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:27:32.820 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:32.830+0000 7f0704a4e640 5 monclient: authenticate success, global_id 4547 2024-03-29T11:27:32.820 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:32.830+0000 7f0704a4e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:27:32.820 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:32.830+0000 7f06fd59a640 1 -- 172.21.15.84:0/1985635217 <== mon.2 v1:172.21.15.84:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f06f0001860 con 0x7f070013f650 2024-03-29T11:27:32.820 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:32.830+0000 7f06fd59a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:27:32.820 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:32.830+0000 7f06fd59a640 1 -- 172.21.15.84:0/1985635217 <== mon.2 v1:172.21.15.84:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4236868694 0 0) 0x7f06f0001cb0 con 0x7f070013f650 2024-03-29T11:27:32.820 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:32.830+0000 7f06fd59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:27:32.820 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:32.830+0000 7f06fd59a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:27:32.820 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:32.830+0000 7f06fd59a640 10 monclient: dump: 2024-03-29T11:27:32.821 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T11:27:32.821 INFO:teuthology.misc.health.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:27:32.821 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T11:18:53.864025+0000 2024-03-29T11:27:32.821 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T11:18:53.864025+0000 2024-03-29T11:27:32.821 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T11:27:32.821 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:27:32.821 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T11:27:32.821 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T11:27:32.821 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T11:27:32.821 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:27:32.821 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:32.830+0000 7f0704a4e640 10 monclient: get_monmap_and_config success 2024-03-29T11:27:32.821 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:32.830+0000 7f0704a4e640 4 set_mon_vals no callback set 2024-03-29T11:27:32.823 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:32.831+0000 7f0704a4e640 10 monclient: shutdown 2024-03-29T11:27:32.823 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:32.831+0000 7f0704a4e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:27:32.823 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:32.831+0000 7f0704a4e640 1 -- 172.21.15.84:0/1985635217 >> v1:172.21.15.84:6790/0 conn(0x7f070013f650 legacy=0x7f07000b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:32.823 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:32.832+0000 7f0704a4e640 1 -- 172.21.15.84:0/1985635217 shutdown_connections 2024-03-29T11:27:32.823 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:32.832+0000 7f0704a4e640 1 -- 172.21.15.84:0/1985635217 >> 172.21.15.84:0/1985635217 conn(0x7f0700066a00 msgr2=0x7f0700066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:27:32.823 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:32.832+0000 7f0704a4e640 1 -- 172.21.15.84:0/1985635217 shutdown_connections 2024-03-29T11:27:32.823 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:32.832+0000 7f0704a4e640 1 -- 172.21.15.84:0/1985635217 wait complete. 2024-03-29T11:27:32.823 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:32.833+0000 7f0704a4e640 10 monclient: build_initial_monmap 2024-03-29T11:27:32.823 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:32.833+0000 7f0704a4e640 10 monclient: monmap: 2024-03-29T11:27:32.823 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-29T11:27:32.823 INFO:teuthology.misc.health.smithi084.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:27:32.823 INFO:teuthology.misc.health.smithi084.stderr:last_changed 0.000000 2024-03-29T11:27:32.823 INFO:teuthology.misc.health.smithi084.stderr:created 0.000000 2024-03-29T11:27:32.823 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-29T11:27:32.824 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:27:32.824 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-29T11:27:32.824 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.noname-b 2024-03-29T11:27:32.824 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-c 2024-03-29T11:27:32.824 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:27:32.824 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:32.834+0000 7f0704a4e640 1 librados: starting msgr at 2024-03-29T11:27:32.824 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:32.834+0000 7f0704a4e640 1 librados: starting objecter 2024-03-29T11:27:32.824 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:32.834+0000 7f0704a4e640 1 Processor -- start 2024-03-29T11:27:32.824 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:32.834+0000 7f0704a4e640 1 -- start start 2024-03-29T11:27:32.824 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:32.834+0000 7f0704a4e640 1 librados: setting wanted keys 2024-03-29T11:27:32.824 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:32.834+0000 7f0704a4e640 1 librados: calling monclient init 2024-03-29T11:27:32.825 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:32.835+0000 7f0704a4e640 10 monclient: init 2024-03-29T11:27:32.827 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:32.835+0000 7f0704a4e640 10 monclient: _reopen_session rank -1 2024-03-29T11:27:32.827 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:32.835+0000 7f0704a4e640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:27:32.827 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:32.836+0000 7f0704a4e640 10 monclient(hunting): picked mon.noname-b con 0x7f07000a9d70 addr v1:172.21.15.179:6789/0 2024-03-29T11:27:32.827 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:32.837+0000 7f0704a4e640 10 monclient(hunting): picked mon.noname-c con 0x7f070013f650 addr v1:172.21.15.84:6790/0 2024-03-29T11:27:32.827 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:32.837+0000 7f0704a4e640 10 monclient(hunting): picked mon.noname-a con 0x7f07001457f0 addr v1:172.21.15.84:6789/0 2024-03-29T11:27:32.827 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:32.837+0000 7f0704a4e640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f07001358b0 con 0x7f07001457f0 2024-03-29T11:27:32.827 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:32.837+0000 7f0704a4e640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f07001414a0 con 0x7f07000a9d70 2024-03-29T11:27:32.827 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:32.837+0000 7f0704a4e640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0700132a30 con 0x7f070013f650 2024-03-29T11:27:32.827 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:32.837+0000 7f0704a4e640 10 monclient(hunting): _renew_subs 2024-03-29T11:27:32.827 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:32.837+0000 7f0704a4e640 10 monclient(hunting): authenticate will time out at 1314.131249s 2024-03-29T11:27:32.827 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:32.837+0000 7f06fdd9b640 1 --1- >> v1:172.21.15.84:6790/0 conn(0x7f070013f650 0x7f070012f900 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.84:6790/0 says I am v1:172.21.15.84:51128/0 (socket says 172.21.15.84:51128) 2024-03-29T11:27:32.828 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:32.837+0000 7f06fdd9b640 1 -- 172.21.15.84:0/841553917 learned_addr learned my addr 172.21.15.84:0/841553917 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-29T11:27:32.828 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:32.838+0000 7f06eeffd640 10 client.?.objecter ms_handle_connect 0x7f070013f650 2024-03-29T11:27:32.828 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:32.838+0000 7f06eeffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:27:32.828 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:32.838+0000 7f06eeffd640 1 -- 172.21.15.84:0/841553917 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 222489249 0 0) 0x7f0700132a30 con 0x7f070013f650 2024-03-29T11:27:32.828 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:32.838+0000 7f06eeffd640 10 monclient(hunting): _negotiate 2024-03-29T11:27:32.828 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:32.838+0000 7f06eeffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:27:32.828 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:32.838+0000 7f06eeffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:27:32.828 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:32.838+0000 7f06eeffd640 10 monclient(hunting): my global_id is 4550 2024-03-29T11:27:32.828 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:32.838+0000 7f06eeffd640 1 -- 172.21.15.84:0/841553917 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f06e0004eb0 con 0x7f070013f650 2024-03-29T11:27:32.834 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:32.840+0000 7f06eeffd640 1 -- 172.21.15.84:0/841553917 <== mon.2 v1:172.21.15.84:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3453745670 0 0) 0x7f06e0004eb0 con 0x7f070013f650 2024-03-29T11:27:32.834 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:32.840+0000 7f06eeffd640 1 -- 172.21.15.84:0/841553917 --> v1:172.21.15.84:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0700132a30 con 0x7f070013f650 2024-03-29T11:27:32.834 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:32.840+0000 7f06eeffd640 1 -- 172.21.15.84:0/841553917 <== mon.2 v1:172.21.15.84:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4236868694 0 0) 0x7f06f0001b70 con 0x7f070013f650 2024-03-29T11:27:32.835 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:32.840+0000 7f06eeffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:27:32.835 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:32.840+0000 7f06eeffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:27:32.835 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:32.840+0000 7f06eeffd640 10 monclient(hunting): dump: 2024-03-29T11:27:32.835 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T11:27:32.835 INFO:teuthology.misc.health.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:27:32.835 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T11:18:53.864025+0000 2024-03-29T11:27:32.835 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T11:18:53.864025+0000 2024-03-29T11:27:32.835 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T11:27:32.835 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:27:32.835 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T11:27:32.835 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T11:27:32.835 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T11:27:32.835 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:27:32.835 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:32.840+0000 7f06eeffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:27:32.835 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:32.841+0000 7f06eeffd640 10 client.?.objecter ms_handle_connect 0x7f07000a9d70 2024-03-29T11:27:32.836 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:32.841+0000 7f06eeffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:27:32.836 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:32.841+0000 7f06eeffd640 1 -- 172.21.15.84:0/841553917 <== mon.2 v1:172.21.15.84:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1331980088 0 0) 0x7f0700132a30 con 0x7f070013f650 2024-03-29T11:27:32.836 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:32.841+0000 7f06eeffd640 1 -- 172.21.15.84:0/841553917 >> v1:172.21.15.179:6789/0 conn(0x7f07000a9d70 legacy=0x7f0700135db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:32.836 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:32.841+0000 7f06eeffd640 1 -- 172.21.15.84:0/841553917 >> v1:172.21.15.84:6789/0 conn(0x7f07001457f0 legacy=0x7f0700130010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:32.836 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:32.842+0000 7f06eeffd640 10 monclient: _finish_hunting 0 2024-03-29T11:27:32.836 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:32.842+0000 7f06eeffd640 1 monclient: found mon.c 2024-03-29T11:27:32.836 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:32.842+0000 7f06eeffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:27:32.836 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:32.842+0000 7f06eeffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-29T11:27:32.836 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:32.842+0000 7f06eeffd640 1 -- 172.21.15.84:0/841553917 --> v1:172.21.15.84:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0700132bc0 con 0x7f070013f650 2024-03-29T11:27:32.836 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:32.842+0000 7f06eeffd640 10 monclient: _finish_auth 0 2024-03-29T11:27:32.836 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:32.842+0000 7f06eeffd640 10 monclient: _check_auth_tickets 2024-03-29T11:27:32.837 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:32.842+0000 7f06eeffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:27:32.837 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:32.842+0000 7f0704a4e640 5 monclient: authenticate success, global_id 4550 2024-03-29T11:27:32.837 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:32.842+0000 7f0704a4e640 10 monclient: _renew_subs 2024-03-29T11:27:32.837 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:32.842+0000 7f0704a4e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-29T11:27:32.837 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:32.842+0000 7f0704a4e640 1 -- 172.21.15.84:0/841553917 --> v1:172.21.15.84:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f07001354b0 con 0x7f070013f650 2024-03-29T11:27:32.837 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:32.842+0000 7f0704a4e640 10 client.4550.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:27:32.837 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:32.842+0000 7f0704a4e640 10 monclient: _renew_subs 2024-03-29T11:27:32.837 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:32.842+0000 7f0704a4e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-29T11:27:32.837 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:32.842+0000 7f0704a4e640 1 -- 172.21.15.84:0/841553917 --> v1:172.21.15.84:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f07001366a0 con 0x7f070013f650 2024-03-29T11:27:32.837 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:32.842+0000 7f0704a4e640 1 librados: init done 2024-03-29T11:27:32.837 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:32.843+0000 7f06eeffd640 1 -- 172.21.15.84:0/841553917 <== mon.2 v1:172.21.15.84:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f06f0004bf0 con 0x7f070013f650 2024-03-29T11:27:32.837 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:32.843+0000 7f06eeffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:27:32.837 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:32.843+0000 7f06eeffd640 1 -- 172.21.15.84:0/841553917 <== mon.2 v1:172.21.15.84:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4236868694 0 0) 0x7f06f0005040 con 0x7f070013f650 2024-03-29T11:27:32.837 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:32.843+0000 7f06eeffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:27:32.838 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:32.843+0000 7f06eeffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:27:32.838 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:32.843+0000 7f06eeffd640 10 monclient: dump: 2024-03-29T11:27:32.838 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T11:27:32.838 INFO:teuthology.misc.health.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:27:32.838 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T11:18:53.864025+0000 2024-03-29T11:27:32.838 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T11:18:53.864025+0000 2024-03-29T11:27:32.838 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T11:27:32.838 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:27:32.838 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T11:27:32.838 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T11:27:32.838 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T11:27:32.838 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:27:32.838 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:32.843+0000 7f06fcd99640 4 set_mon_vals no callback set 2024-03-29T11:27:32.838 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:32.844+0000 7f0704a4e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:27:32.839 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:32.846+0000 7f0704a4e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:27:32.839 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:32.846+0000 7f0704a4e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-29T11:27:32.839 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:32.846+0000 7f0704a4e640 1 -- 172.21.15.84:0/841553917 --> v1:172.21.15.84:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f06e4005080 con 0x7f070013f650 2024-03-29T11:27:32.839 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:32.846+0000 7f06eeffd640 1 -- 172.21.15.84:0/841553917 <== mon.2 v1:172.21.15.84:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4033942909 0 0) 0x7f06f00051a0 con 0x7f070013f650 2024-03-29T11:27:32.839 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:32.846+0000 7f06eeffd640 1 -- 172.21.15.84:0/841553917 <== mon.2 v1:172.21.15.84:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2808139587 0 0) 0x7f06f0084cc0 con 0x7f070013f650 2024-03-29T11:27:32.839 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:32.846+0000 7f06eeffd640 10 client.4550.objecter ms_dispatch 0x7f0700000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:27:32.839 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:32.846+0000 7f06eeffd640 3 client.4550.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:27:32.839 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:32.846+0000 7f06eeffd640 3 client.4550.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:27:32.839 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:32.846+0000 7f06eeffd640 20 client.4550.objecter dump_active .. 0 homeless 2024-03-29T11:27:32.839 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:32.846+0000 7f06fe59c640 10 monclient: get_auth_request con 0x7f06e006dc60 auth_method 0 2024-03-29T11:27:32.839 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:32.847+0000 7f06eeffd640 10 client.4550.objecter ms_handle_connect 0x7f06e006dc60 2024-03-29T11:27:32.843 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:32.854+0000 7f06eeffd640 1 -- 172.21.15.84:0/841553917 <== mon.2 v1:172.21.15.84:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f06f00505a0 con 0x7f070013f650 2024-03-29T11:27:32.843 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:32.854+0000 7f06eeffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:27:32.843 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:32.854+0000 7f06eeffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:27:33.170 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:33.181+0000 7f0704a4e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:27:33.170 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:33.181+0000 7f0704a4e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:27:33.170 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:33.181+0000 7f0704a4e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-29T11:27:33.170 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:33.181+0000 7f0704a4e640 1 -- 172.21.15.84:0/841553917 --> v1:172.21.15.84:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f06e4005330 con 0x7f070013f650 2024-03-29T11:27:33.174 INFO:teuthology.misc.health.smithi084.stdout: 2024-03-29T11:27:33.174 INFO:teuthology.misc.health.smithi084.stdout:{"epoch":10,"fsid":"2c3e002f-9eb0-4860-853b-4ef1ef562213","created":"2024-03-29T11:19:32.029169+0000","modified":"2024-03-29T11:19:34.200299+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:34.200299+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"5c39e276-2034-4716-bbc0-57a634eefa92","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"cd580e99-109f-4bf1-88c0-a5987bc8b9ed","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"11e60a5a-051d-453f-a190-66387cb35592","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"2c329fbe-bbc4-4030-bf87-d711cad04679","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f4213d4a-bd36-4433-91ae-d7e2b02481d3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"f0cf44ea-6055-48af-805b-aed858102ff9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"d895eaa0-cc0b-4ee4-9b5a-2dfcd89a60b7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"9bc5a7bc-8f52-4db9-96d6-de5e68d52eaa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:27:33.174 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:33.185+0000 7f06eeffd640 1 -- 172.21.15.84:0/841553917 <== mon.2 v1:172.21.15.84:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4209477549) 0x7f06f00524d0 con 0x7f070013f650 2024-03-29T11:27:33.174 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:33.185+0000 7f06eeffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:27:33.174 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:33.185+0000 7f06eeffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:27:33.177 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:33.189+0000 7f0704a4e640 10 librados: watch_flush enter 2024-03-29T11:27:33.177 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:33.189+0000 7f0704a4e640 10 librados: watch_flush exit 2024-03-29T11:27:33.177 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:33.189+0000 7f0704a4e640 20 client.4550.objecter shutdown clearing up homeless session... 2024-03-29T11:27:33.177 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:33.189+0000 7f0704a4e640 10 client.4550.objecter successfully canceled tick 2024-03-29T11:27:33.179 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:33.191+0000 7f0704a4e640 1 -- 172.21.15.84:0/841553917 >> v1:172.21.15.179:6816/2981134980 conn(0x7f06e006dc60 legacy=0x7f06e0070120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:33.179 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:33.191+0000 7f0704a4e640 10 monclient: shutdown 2024-03-29T11:27:33.179 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:33.191+0000 7f0704a4e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:27:33.179 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:33.191+0000 7f0704a4e640 1 -- 172.21.15.84:0/841553917 >> v1:172.21.15.84:6790/0 conn(0x7f070013f650 legacy=0x7f070012f900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:33.181 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:33.193+0000 7f0704a4e640 1 -- 172.21.15.84:0/841553917 shutdown_connections 2024-03-29T11:27:33.181 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:33.193+0000 7f0704a4e640 1 -- 172.21.15.84:0/841553917 >> 172.21.15.84:0/841553917 conn(0x7f0700066a00 msgr2=0x7f07000b3a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:27:33.181 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:33.193+0000 7f0704a4e640 1 -- 172.21.15.84:0/841553917 shutdown_connections 2024-03-29T11:27:33.181 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:33.193+0000 7f0704a4e640 1 -- 172.21.15.84:0/841553917 wait complete. 2024-03-29T11:27:33.182 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:33.193+0000 7f0704a4e640 1 librados: shutdown 2024-03-29T11:27:33.198 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:27:40.010 DEBUG:teuthology.orchestra.run.smithi084:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:27:40.158 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:40.169+0000 7f7eb89e5640 10 monclient: get_monmap_and_config 2024-03-29T11:27:40.158 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:40.169+0000 7f7eb89e5640 10 monclient: build_initial_monmap 2024-03-29T11:27:40.158 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:40.169+0000 7f7eb89e5640 10 monclient: monmap: 2024-03-29T11:27:40.159 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-29T11:27:40.159 INFO:teuthology.misc.health.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:27:40.159 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T11:27:40.170724+0000 2024-03-29T11:27:40.159 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T11:27:40.170724+0000 2024-03-29T11:27:40.159 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-29T11:27:40.159 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:27:40.159 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-29T11:27:40.159 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.noname-c 2024-03-29T11:27:40.159 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-b 2024-03-29T11:27:40.159 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:27:40.161 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:40.172+0000 7f7eb89e5640 1 Processor -- start 2024-03-29T11:27:40.161 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:40.172+0000 7f7eb89e5640 1 -- start start 2024-03-29T11:27:40.161 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:40.172+0000 7f7eb89e5640 10 monclient: init 2024-03-29T11:27:40.161 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:40.173+0000 7f7eb89e5640 10 monclient: _reopen_session rank -1 2024-03-29T11:27:40.161 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:40.173+0000 7f7eb89e5640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:27:40.161 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:40.173+0000 7f7eb89e5640 10 monclient(hunting): picked mon.noname-c con 0x7f7eac02b4c0 addr v1:172.21.15.179:6789/0 2024-03-29T11:27:40.161 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:40.173+0000 7f7eb89e5640 10 monclient(hunting): picked mon.noname-a con 0x7f7eac00b8a0 addr v1:172.21.15.84:6789/0 2024-03-29T11:27:40.161 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:40.173+0000 7f7eb89e5640 10 monclient(hunting): picked mon.noname-b con 0x7f7eac02a5a0 addr v1:172.21.15.84:6790/0 2024-03-29T11:27:40.162 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:40.173+0000 7f7eb89e5640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7eac0c11a0 con 0x7f7eac00b8a0 2024-03-29T11:27:40.162 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:40.173+0000 7f7eb89e5640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7eac02c830 con 0x7f7eac02b4c0 2024-03-29T11:27:40.162 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:40.173+0000 7f7eb89e5640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7eac02afa0 con 0x7f7eac02a5a0 2024-03-29T11:27:40.162 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:40.173+0000 7f7eb89e5640 10 monclient(hunting): _renew_subs 2024-03-29T11:27:40.162 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:40.173+0000 7f7eb89e5640 10 monclient(hunting): authenticate will time out at 1321.467254s 2024-03-29T11:27:40.162 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:40.174+0000 7f7eb359e640 1 --1- >> v1:172.21.15.84:6790/0 conn(0x7f7eac02a5a0 0x7f7eac0c6d40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.84:6790/0 says I am v1:172.21.15.84:57092/0 (socket says 172.21.15.84:57092) 2024-03-29T11:27:40.162 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:40.174+0000 7f7eb359e640 1 -- 172.21.15.84:0/1677191341 learned_addr learned my addr 172.21.15.84:0/1677191341 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-29T11:27:40.163 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:40.174+0000 7f7eb1d9b640 1 -- 172.21.15.84:0/1677191341 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1517746614 0 0) 0x7f7eac02afa0 con 0x7f7eac02a5a0 2024-03-29T11:27:40.163 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:40.174+0000 7f7eb1d9b640 10 monclient(hunting): _negotiate 2024-03-29T11:27:40.163 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:40.174+0000 7f7eb1d9b640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:27:40.163 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:40.174+0000 7f7eb1d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:27:40.163 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:40.174+0000 7f7eb1d9b640 10 monclient(hunting): my global_id is 4553 2024-03-29T11:27:40.163 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:40.174+0000 7f7eb1d9b640 1 -- 172.21.15.84:0/1677191341 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7e9c003520 con 0x7f7eac02a5a0 2024-03-29T11:27:40.165 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:40.177+0000 7f7eb1d9b640 1 -- 172.21.15.84:0/1677191341 <== mon.2 v1:172.21.15.84:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2264865334 0 0) 0x7f7e9c003520 con 0x7f7eac02a5a0 2024-03-29T11:27:40.165 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:40.177+0000 7f7eb1d9b640 1 -- 172.21.15.84:0/1677191341 >> v1:172.21.15.179:6789/0 conn(0x7f7eac02b4c0 legacy=0x7f7eac006890 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-29T11:27:40.165 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:40.177+0000 7f7eb1d9b640 1 -- 172.21.15.84:0/1677191341 >> v1:172.21.15.84:6789/0 conn(0x7f7eac00b8a0 legacy=0x7f7eac0c0a90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:27:40.166 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:40.177+0000 7f7eb1d9b640 10 monclient: _finish_hunting 0 2024-03-29T11:27:40.166 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:40.177+0000 7f7eb1d9b640 1 monclient: found mon.noname-b 2024-03-29T11:27:40.166 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:40.177+0000 7f7eb1d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:27:40.166 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:40.177+0000 7f7eb1d9b640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.84:6790/0 2024-03-29T11:27:40.166 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:40.177+0000 7f7eb1d9b640 1 -- 172.21.15.84:0/1677191341 --> v1:172.21.15.84:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7eac02aba0 con 0x7f7eac02a5a0 2024-03-29T11:27:40.166 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:40.177+0000 7f7eb1d9b640 10 monclient: _finish_auth 0 2024-03-29T11:27:40.166 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:40.177+0000 7f7eb1d9b640 10 monclient: _check_auth_tickets 2024-03-29T11:27:40.166 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:40.177+0000 7f7eb1d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:27:40.167 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:40.177+0000 7f7eb1d9b640 1 -- 172.21.15.84:0/1677191341 <== mon.2 v1:172.21.15.84:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4236868694 0 0) 0x7f7ea4001690 con 0x7f7eac02a5a0 2024-03-29T11:27:40.167 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:40.177+0000 7f7eb1d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:27:40.167 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:40.177+0000 7f7eb1d9b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:27:40.167 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:40.177+0000 7f7eb1d9b640 10 monclient: dump: 2024-03-29T11:27:40.167 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T11:27:40.167 INFO:teuthology.misc.health.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:27:40.167 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T11:18:53.864025+0000 2024-03-29T11:27:40.167 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T11:18:53.864025+0000 2024-03-29T11:27:40.167 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T11:27:40.167 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:27:40.167 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T11:27:40.167 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T11:27:40.167 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T11:27:40.167 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:27:40.168 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:40.177+0000 7f7eb89e5640 5 monclient: authenticate success, global_id 4553 2024-03-29T11:27:40.168 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:40.177+0000 7f7eb89e5640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:27:40.168 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:40.177+0000 7f7eb1d9b640 1 -- 172.21.15.84:0/1677191341 <== mon.2 v1:172.21.15.84:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7ea4001920 con 0x7f7eac02a5a0 2024-03-29T11:27:40.168 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:40.177+0000 7f7eb1d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:27:40.168 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:40.177+0000 7f7eb1d9b640 1 -- 172.21.15.84:0/1677191341 <== mon.2 v1:172.21.15.84:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4236868694 0 0) 0x7f7ea4001cb0 con 0x7f7eac02a5a0 2024-03-29T11:27:40.168 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:40.177+0000 7f7eb89e5640 10 monclient: get_monmap_and_config success 2024-03-29T11:27:40.168 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:40.177+0000 7f7eb89e5640 4 set_mon_vals no callback set 2024-03-29T11:27:40.168 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:40.178+0000 7f7eb1d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:27:40.168 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:40.178+0000 7f7eb1d9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:27:40.168 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:40.178+0000 7f7eb1d9b640 10 monclient: dump: 2024-03-29T11:27:40.168 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T11:27:40.168 INFO:teuthology.misc.health.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:27:40.168 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T11:18:53.864025+0000 2024-03-29T11:27:40.168 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T11:18:53.864025+0000 2024-03-29T11:27:40.168 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T11:27:40.169 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:27:40.169 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T11:27:40.169 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T11:27:40.169 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T11:27:40.169 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:27:40.170 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:40.179+0000 7f7eb89e5640 10 monclient: shutdown 2024-03-29T11:27:40.170 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:40.179+0000 7f7eb89e5640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:27:40.170 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:40.179+0000 7f7eb89e5640 1 -- 172.21.15.84:0/1677191341 >> v1:172.21.15.84:6790/0 conn(0x7f7eac02a5a0 legacy=0x7f7eac0c6d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:40.170 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:40.180+0000 7f7eb89e5640 1 -- 172.21.15.84:0/1677191341 shutdown_connections 2024-03-29T11:27:40.170 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:40.180+0000 7f7eb89e5640 1 -- 172.21.15.84:0/1677191341 >> 172.21.15.84:0/1677191341 conn(0x7f7eac0058f0 msgr2=0x7f7eac005d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:27:40.170 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:40.180+0000 7f7eb89e5640 1 -- 172.21.15.84:0/1677191341 shutdown_connections 2024-03-29T11:27:40.170 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:40.180+0000 7f7eb89e5640 1 -- 172.21.15.84:0/1677191341 wait complete. 2024-03-29T11:27:40.171 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:40.181+0000 7f7eb89e5640 10 monclient: build_initial_monmap 2024-03-29T11:27:40.171 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:40.182+0000 7f7eb89e5640 10 monclient: monmap: 2024-03-29T11:27:40.171 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-29T11:27:40.171 INFO:teuthology.misc.health.smithi084.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:27:40.171 INFO:teuthology.misc.health.smithi084.stderr:last_changed 0.000000 2024-03-29T11:27:40.171 INFO:teuthology.misc.health.smithi084.stderr:created 0.000000 2024-03-29T11:27:40.171 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-29T11:27:40.171 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:27:40.171 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-29T11:27:40.171 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.noname-b 2024-03-29T11:27:40.171 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-c 2024-03-29T11:27:40.171 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:27:40.173 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:40.182+0000 7f7eb89e5640 1 librados: starting msgr at 2024-03-29T11:27:40.173 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:40.182+0000 7f7eb89e5640 1 librados: starting objecter 2024-03-29T11:27:40.173 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:40.183+0000 7f7eb89e5640 1 Processor -- start 2024-03-29T11:27:40.173 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:40.183+0000 7f7eb89e5640 1 -- start start 2024-03-29T11:27:40.173 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:40.183+0000 7f7eb89e5640 1 librados: setting wanted keys 2024-03-29T11:27:40.173 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:40.183+0000 7f7eb89e5640 1 librados: calling monclient init 2024-03-29T11:27:40.173 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:40.183+0000 7f7eb89e5640 10 monclient: init 2024-03-29T11:27:40.174 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:40.184+0000 7f7eb89e5640 10 monclient: _reopen_session rank -1 2024-03-29T11:27:40.174 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:40.184+0000 7f7eb89e5640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:27:40.174 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:40.185+0000 7f7eb89e5640 10 monclient(hunting): picked mon.noname-c con 0x7f7eac00b8a0 addr v1:172.21.15.84:6790/0 2024-03-29T11:27:40.175 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:40.185+0000 7f7eb89e5640 10 monclient(hunting): picked mon.noname-a con 0x7f7eac02a5a0 addr v1:172.21.15.84:6789/0 2024-03-29T11:27:40.175 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:40.186+0000 7f7eb89e5640 10 monclient(hunting): picked mon.noname-b con 0x7f7eac02b4c0 addr v1:172.21.15.179:6789/0 2024-03-29T11:27:40.175 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:40.186+0000 7f7eb89e5640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7eac15d820 con 0x7f7eac02a5a0 2024-03-29T11:27:40.176 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:40.186+0000 7f7eb89e5640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7eac15d350 con 0x7f7eac02b4c0 2024-03-29T11:27:40.176 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:40.186+0000 7f7eb89e5640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7eac15d4e0 con 0x7f7eac00b8a0 2024-03-29T11:27:40.176 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:40.186+0000 7f7eb89e5640 10 monclient(hunting): _renew_subs 2024-03-29T11:27:40.176 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:40.186+0000 7f7eb89e5640 10 monclient(hunting): authenticate will time out at 1321.480945s 2024-03-29T11:27:40.176 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:40.186+0000 7f7eb2d9d640 1 --1- >> v1:172.21.15.84:6790/0 conn(0x7f7eac00b8a0 0x7f7eac15cc40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.84:6790/0 says I am v1:172.21.15.84:58694/0 (socket says 172.21.15.84:58694) 2024-03-29T11:27:40.176 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:40.187+0000 7f7eb2d9d640 1 -- 172.21.15.84:0/3775018029 learned_addr learned my addr 172.21.15.84:0/3775018029 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-29T11:27:40.176 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:40.187+0000 7f7e9b7fe640 10 client.?.objecter ms_handle_connect 0x7f7eac00b8a0 2024-03-29T11:27:40.176 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:40.187+0000 7f7e9b7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:27:40.176 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:40.187+0000 7f7e9b7fe640 1 -- 172.21.15.84:0/3775018029 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1835683754 0 0) 0x7f7eac15d4e0 con 0x7f7eac00b8a0 2024-03-29T11:27:40.176 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:40.187+0000 7f7e9b7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:27:40.176 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:40.187+0000 7f7e9b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:27:40.176 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:40.187+0000 7f7e9b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:27:40.176 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:40.187+0000 7f7e9b7fe640 10 monclient(hunting): my global_id is 4556 2024-03-29T11:27:40.177 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:40.187+0000 7f7e9b7fe640 1 -- 172.21.15.84:0/3775018029 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7ea0004eb0 con 0x7f7eac00b8a0 2024-03-29T11:27:40.177 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:40.188+0000 7f7e9b7fe640 1 -- 172.21.15.84:0/3775018029 <== mon.2 v1:172.21.15.84:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1197796700 0 0) 0x7f7ea0004eb0 con 0x7f7eac00b8a0 2024-03-29T11:27:40.178 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:40.188+0000 7f7e9b7fe640 1 -- 172.21.15.84:0/3775018029 --> v1:172.21.15.84:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7eac15d4e0 con 0x7f7eac00b8a0 2024-03-29T11:27:40.178 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:40.188+0000 7f7e9b7fe640 1 -- 172.21.15.84:0/3775018029 <== mon.2 v1:172.21.15.84:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4236868694 0 0) 0x7f7eb404fdd0 con 0x7f7eac00b8a0 2024-03-29T11:27:40.178 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:40.188+0000 7f7e9b7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:27:40.178 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:40.188+0000 7f7e9b7fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:27:40.178 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:40.188+0000 7f7e9b7fe640 10 monclient(hunting): dump: 2024-03-29T11:27:40.178 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T11:27:40.178 INFO:teuthology.misc.health.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:27:40.178 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T11:18:53.864025+0000 2024-03-29T11:27:40.178 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T11:18:53.864025+0000 2024-03-29T11:27:40.178 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T11:27:40.178 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:27:40.178 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T11:27:40.178 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T11:27:40.179 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T11:27:40.179 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:27:40.179 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:40.188+0000 7f7e9b7fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:27:40.179 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:40.188+0000 7f7e9b7fe640 1 -- 172.21.15.84:0/3775018029 <== mon.2 v1:172.21.15.84:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1898525518 0 0) 0x7f7eac15d4e0 con 0x7f7eac00b8a0 2024-03-29T11:27:40.179 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:40.188+0000 7f7e9b7fe640 1 -- 172.21.15.84:0/3775018029 >> v1:172.21.15.179:6789/0 conn(0x7f7eac02b4c0 legacy=0x7f7eac159740 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:27:40.185 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:40.190+0000 7f7e9b7fe640 1 -- 172.21.15.84:0/3775018029 >> v1:172.21.15.84:6789/0 conn(0x7f7eac02a5a0 legacy=0x7f7eac0b4eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:40.185 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:40.190+0000 7f7e9b7fe640 10 monclient: _finish_hunting 0 2024-03-29T11:27:40.185 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:40.190+0000 7f7e9b7fe640 1 monclient: found mon.c 2024-03-29T11:27:40.185 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:40.190+0000 7f7e9b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:27:40.185 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:40.190+0000 7f7e9b7fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-29T11:27:40.185 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:40.190+0000 7f7e9b7fe640 1 -- 172.21.15.84:0/3775018029 --> v1:172.21.15.84:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7eac0b6810 con 0x7f7eac00b8a0 2024-03-29T11:27:40.185 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:40.190+0000 7f7e9b7fe640 10 monclient: _finish_auth 0 2024-03-29T11:27:40.185 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:40.190+0000 7f7e9b7fe640 10 monclient: _check_auth_tickets 2024-03-29T11:27:40.185 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:40.190+0000 7f7e9b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:27:40.185 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:40.190+0000 7f7eb89e5640 5 monclient: authenticate success, global_id 4556 2024-03-29T11:27:40.185 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:40.190+0000 7f7eb89e5640 10 monclient: _renew_subs 2024-03-29T11:27:40.186 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:40.190+0000 7f7eb89e5640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-29T11:27:40.186 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:40.190+0000 7f7eb89e5640 1 -- 172.21.15.84:0/3775018029 --> v1:172.21.15.84:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7eac0b6a60 con 0x7f7eac00b8a0 2024-03-29T11:27:40.186 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:40.190+0000 7f7eb89e5640 10 client.4556.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:27:40.186 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:40.190+0000 7f7eb89e5640 10 monclient: _renew_subs 2024-03-29T11:27:40.186 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:40.190+0000 7f7eb89e5640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-29T11:27:40.186 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:40.190+0000 7f7eb89e5640 1 -- 172.21.15.84:0/3775018029 --> v1:172.21.15.84:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f7eac0b57a0 con 0x7f7eac00b8a0 2024-03-29T11:27:40.186 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:40.190+0000 7f7eb89e5640 1 librados: init done 2024-03-29T11:27:40.186 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:40.190+0000 7f7e9b7fe640 1 -- 172.21.15.84:0/3775018029 <== mon.2 v1:172.21.15.84:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7eb404f400 con 0x7f7eac00b8a0 2024-03-29T11:27:40.186 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:40.190+0000 7f7e9b7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:27:40.186 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:40.190+0000 7f7e9b7fe640 1 -- 172.21.15.84:0/3775018029 <== mon.2 v1:172.21.15.84:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4236868694 0 0) 0x7f7eb4075710 con 0x7f7eac00b8a0 2024-03-29T11:27:40.186 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:40.190+0000 7f7e9b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:27:40.186 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:40.190+0000 7f7e9b7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:27:40.186 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:40.190+0000 7f7e9b7fe640 10 monclient: dump: 2024-03-29T11:27:40.186 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T11:27:40.187 INFO:teuthology.misc.health.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:27:40.187 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T11:18:53.864025+0000 2024-03-29T11:27:40.187 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T11:18:53.864025+0000 2024-03-29T11:27:40.187 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T11:27:40.187 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:27:40.187 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T11:27:40.187 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T11:27:40.188 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T11:27:40.188 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:27:40.188 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:40.190+0000 7f7eb0d99640 4 set_mon_vals no callback set 2024-03-29T11:27:40.188 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:40.192+0000 7f7e9b7fe640 1 -- 172.21.15.84:0/3775018029 <== mon.2 v1:172.21.15.84:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4033942909 0 0) 0x7f7eb4067bf0 con 0x7f7eac00b8a0 2024-03-29T11:27:40.188 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:40.192+0000 7f7eb89e5640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:27:40.188 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:40.192+0000 7f7eb89e5640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:27:40.188 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:40.192+0000 7f7eb89e5640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-29T11:27:40.188 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:40.192+0000 7f7eb89e5640 1 -- 172.21.15.84:0/3775018029 --> v1:172.21.15.84:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7e94005080 con 0x7f7eac00b8a0 2024-03-29T11:27:40.188 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:40.193+0000 7f7e9b7fe640 1 -- 172.21.15.84:0/3775018029 <== mon.2 v1:172.21.15.84:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2808139587 0 0) 0x7f7eb40ff610 con 0x7f7eac00b8a0 2024-03-29T11:27:40.188 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:40.193+0000 7f7e9b7fe640 10 client.4556.objecter ms_dispatch 0x7f7eb4000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:27:40.188 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:40.193+0000 7f7e9b7fe640 3 client.4556.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:27:40.188 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:40.193+0000 7f7e9b7fe640 3 client.4556.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:27:40.188 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:40.193+0000 7f7e9b7fe640 20 client.4556.objecter dump_active .. 0 homeless 2024-03-29T11:27:40.188 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:40.193+0000 7f7eb259c640 10 monclient: get_auth_request con 0x7f7ea006dad0 auth_method 0 2024-03-29T11:27:40.189 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:40.194+0000 7f7e9b7fe640 10 client.4556.objecter ms_handle_connect 0x7f7ea006dad0 2024-03-29T11:27:40.190 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:40.200+0000 7f7e9b7fe640 1 -- 172.21.15.84:0/3775018029 <== mon.2 v1:172.21.15.84:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f7eb40cafb0 con 0x7f7eac00b8a0 2024-03-29T11:27:40.190 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:40.200+0000 7f7e9b7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:27:40.190 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:40.200+0000 7f7e9b7fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:27:40.512 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:40.523+0000 7f7eb89e5640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:27:40.512 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:40.523+0000 7f7eb89e5640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:27:40.512 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:40.523+0000 7f7eb89e5640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-29T11:27:40.512 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:40.523+0000 7f7eb89e5640 1 -- 172.21.15.84:0/3775018029 --> v1:172.21.15.84:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7e94005330 con 0x7f7eac00b8a0 2024-03-29T11:27:40.514 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:40.525+0000 7f7e9b7fe640 1 -- 172.21.15.84:0/3775018029 <== mon.2 v1:172.21.15.84:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4209477549) 0x7f7eb40fecc0 con 0x7f7eac00b8a0 2024-03-29T11:27:40.514 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:40.526+0000 7f7e9b7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:27:40.514 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:40.526+0000 7f7e9b7fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:27:40.515 INFO:teuthology.misc.health.smithi084.stdout: 2024-03-29T11:27:40.515 INFO:teuthology.misc.health.smithi084.stdout:{"epoch":10,"fsid":"2c3e002f-9eb0-4860-853b-4ef1ef562213","created":"2024-03-29T11:19:32.029169+0000","modified":"2024-03-29T11:19:34.200299+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:34.200299+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"5c39e276-2034-4716-bbc0-57a634eefa92","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"cd580e99-109f-4bf1-88c0-a5987bc8b9ed","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"11e60a5a-051d-453f-a190-66387cb35592","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"2c329fbe-bbc4-4030-bf87-d711cad04679","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f4213d4a-bd36-4433-91ae-d7e2b02481d3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"f0cf44ea-6055-48af-805b-aed858102ff9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"d895eaa0-cc0b-4ee4-9b5a-2dfcd89a60b7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"9bc5a7bc-8f52-4db9-96d6-de5e68d52eaa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:27:40.518 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:40.529+0000 7f7eb89e5640 10 librados: watch_flush enter 2024-03-29T11:27:40.518 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:40.529+0000 7f7eb89e5640 10 librados: watch_flush exit 2024-03-29T11:27:40.518 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:40.529+0000 7f7eb89e5640 20 client.4556.objecter shutdown clearing up homeless session... 2024-03-29T11:27:40.518 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:40.529+0000 7f7eb89e5640 10 client.4556.objecter successfully canceled tick 2024-03-29T11:27:40.518 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:40.529+0000 7f7eb89e5640 1 -- 172.21.15.84:0/3775018029 >> v1:172.21.15.179:6816/2981134980 conn(0x7f7ea006dad0 legacy=0x7f7ea006ff90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:40.518 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:40.529+0000 7f7eb89e5640 10 monclient: shutdown 2024-03-29T11:27:40.519 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:40.529+0000 7f7eb89e5640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:27:40.519 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:40.529+0000 7f7eb89e5640 1 -- 172.21.15.84:0/3775018029 >> v1:172.21.15.84:6790/0 conn(0x7f7eac00b8a0 legacy=0x7f7eac15cc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:40.519 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:40.530+0000 7f7eb89e5640 1 -- 172.21.15.84:0/3775018029 shutdown_connections 2024-03-29T11:27:40.519 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:40.530+0000 7f7eb89e5640 1 -- 172.21.15.84:0/3775018029 >> 172.21.15.84:0/3775018029 conn(0x7f7eac0058f0 msgr2=0x7f7eac0d0a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:27:40.520 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:40.532+0000 7f7eb89e5640 1 -- 172.21.15.84:0/3775018029 shutdown_connections 2024-03-29T11:27:40.520 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:40.532+0000 7f7eb89e5640 1 -- 172.21.15.84:0/3775018029 wait complete. 2024-03-29T11:27:40.520 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:40.532+0000 7f7eb89e5640 1 librados: shutdown 2024-03-29T11:27:40.540 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:27:47.347 DEBUG:teuthology.orchestra.run.smithi084:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:27:47.501 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:47.512+0000 7f4d34b72640 10 monclient: get_monmap_and_config 2024-03-29T11:27:47.501 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:47.512+0000 7f4d34b72640 10 monclient: build_initial_monmap 2024-03-29T11:27:47.501 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:47.512+0000 7f4d34b72640 10 monclient: monmap: 2024-03-29T11:27:47.501 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-29T11:27:47.501 INFO:teuthology.misc.health.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:27:47.501 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T11:27:47.513519+0000 2024-03-29T11:27:47.501 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T11:27:47.513519+0000 2024-03-29T11:27:47.501 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-29T11:27:47.502 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:27:47.502 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-29T11:27:47.502 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.noname-c 2024-03-29T11:27:47.502 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-b 2024-03-29T11:27:47.502 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:27:47.504 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:47.513+0000 7f4d34b72640 1 Processor -- start 2024-03-29T11:27:47.504 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:47.513+0000 7f4d34b72640 1 -- start start 2024-03-29T11:27:47.504 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:47.513+0000 7f4d34b72640 10 monclient: init 2024-03-29T11:27:47.505 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:47.514+0000 7f4d34b72640 10 monclient: _reopen_session rank -1 2024-03-29T11:27:47.505 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:47.514+0000 7f4d34b72640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:27:47.505 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:47.514+0000 7f4d34b72640 10 monclient(hunting): picked mon.noname-a con 0x7f4d300b0860 addr v1:172.21.15.84:6789/0 2024-03-29T11:27:47.505 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:47.514+0000 7f4d34b72640 10 monclient(hunting): picked mon.noname-b con 0x7f4d3013ef30 addr v1:172.21.15.84:6790/0 2024-03-29T11:27:47.505 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:47.514+0000 7f4d34b72640 10 monclient(hunting): picked mon.noname-c con 0x7f4d300af8e0 addr v1:172.21.15.179:6789/0 2024-03-29T11:27:47.505 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:47.514+0000 7f4d34b72640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4d300b0c40 con 0x7f4d300b0860 2024-03-29T11:27:47.505 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:47.514+0000 7f4d34b72640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4d300aed10 con 0x7f4d300af8e0 2024-03-29T11:27:47.505 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:47.514+0000 7f4d34b72640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4d300afd50 con 0x7f4d3013ef30 2024-03-29T11:27:47.505 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:47.514+0000 7f4d34b72640 10 monclient(hunting): _renew_subs 2024-03-29T11:27:47.505 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:47.514+0000 7f4d34b72640 10 monclient(hunting): authenticate will time out at 1328.808723s 2024-03-29T11:27:47.505 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:47.516+0000 7f4d2dd9b640 1 --1- >> v1:172.21.15.84:6790/0 conn(0x7f4d3013ef30 0x7f4d300af1d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.84:6790/0 says I am v1:172.21.15.84:33512/0 (socket says 172.21.15.84:33512) 2024-03-29T11:27:47.506 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:47.516+0000 7f4d2dd9b640 1 -- 172.21.15.84:0/208326235 learned_addr learned my addr 172.21.15.84:0/208326235 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-29T11:27:47.509 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:47.518+0000 7f4d2d59a640 1 -- 172.21.15.84:0/208326235 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1676423850 0 0) 0x7f4d300afd50 con 0x7f4d3013ef30 2024-03-29T11:27:47.509 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:47.518+0000 7f4d2d59a640 10 monclient(hunting): _negotiate 2024-03-29T11:27:47.509 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:47.518+0000 7f4d2d59a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:27:47.509 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:47.518+0000 7f4d2d59a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:27:47.509 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:47.518+0000 7f4d2d59a640 10 monclient(hunting): my global_id is 4559 2024-03-29T11:27:47.509 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:47.518+0000 7f4d2d59a640 1 -- 172.21.15.84:0/208326235 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4d18003520 con 0x7f4d3013ef30 2024-03-29T11:27:47.509 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:47.519+0000 7f4d2ed9d640 1 -- 172.21.15.84:0/208326235 >> v1:172.21.15.179:6789/0 conn(0x7f4d300af8e0 legacy=0x7f4d30149a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.179:6789/0 2024-03-29T11:27:47.509 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:47.519+0000 7f4d2d59a640 1 -- 172.21.15.84:0/208326235 <== mon.0 v1:172.21.15.84:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3034874227 0 0) 0x7f4d300b0c40 con 0x7f4d300b0860 2024-03-29T11:27:47.509 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:47.519+0000 7f4d2d59a640 10 monclient(hunting): _negotiate 2024-03-29T11:27:47.509 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:47.519+0000 7f4d2d59a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:27:47.509 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:47.519+0000 7f4d2d59a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:27:47.509 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:47.519+0000 7f4d2d59a640 10 monclient(hunting): my global_id is 4515 2024-03-29T11:27:47.510 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:47.519+0000 7f4d2d59a640 1 -- 172.21.15.84:0/208326235 --> v1:172.21.15.84:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4d18003840 con 0x7f4d300b0860 2024-03-29T11:27:47.510 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:47.520+0000 7f4d2d59a640 1 -- 172.21.15.84:0/208326235 <== mon.0 v1:172.21.15.84:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1613749211 0 0) 0x7f4d18003840 con 0x7f4d300b0860 2024-03-29T11:27:47.510 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:47.520+0000 7f4d2d59a640 1 -- 172.21.15.84:0/208326235 >> v1:172.21.15.84:6790/0 conn(0x7f4d3013ef30 legacy=0x7f4d300af1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:47.510 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:47.520+0000 7f4d2d59a640 1 -- 172.21.15.84:0/208326235 >> v1:172.21.15.179:6789/0 conn(0x7f4d300af8e0 legacy=0x7f4d30149a70 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:27:47.510 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:47.520+0000 7f4d2d59a640 10 monclient: _finish_hunting 0 2024-03-29T11:27:47.510 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:47.520+0000 7f4d2d59a640 1 monclient: found mon.noname-a 2024-03-29T11:27:47.511 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:47.520+0000 7f4d2d59a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:27:47.511 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:47.520+0000 7f4d2d59a640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.84:6789/0 2024-03-29T11:27:47.511 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:47.520+0000 7f4d2d59a640 1 -- 172.21.15.84:0/208326235 --> v1:172.21.15.84:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d300ab1b0 con 0x7f4d300b0860 2024-03-29T11:27:47.511 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:47.520+0000 7f4d2d59a640 10 monclient: _finish_auth 0 2024-03-29T11:27:47.511 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:47.520+0000 7f4d2d59a640 10 monclient: _check_auth_tickets 2024-03-29T11:27:47.511 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:47.520+0000 7f4d2d59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:27:47.512 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:47.520+0000 7f4d2d59a640 1 -- 172.21.15.84:0/208326235 <== mon.0 v1:172.21.15.84:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4236868694 0 0) 0x7f4d24001670 con 0x7f4d300b0860 2024-03-29T11:27:47.512 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:47.520+0000 7f4d2d59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:27:47.512 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:47.520+0000 7f4d2d59a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:27:47.512 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:47.520+0000 7f4d2d59a640 10 monclient: dump: 2024-03-29T11:27:47.512 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T11:27:47.512 INFO:teuthology.misc.health.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:27:47.512 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T11:18:53.864025+0000 2024-03-29T11:27:47.512 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T11:18:53.864025+0000 2024-03-29T11:27:47.512 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T11:27:47.512 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:27:47.513 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T11:27:47.513 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T11:27:47.513 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T11:27:47.513 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:27:47.513 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:47.520+0000 7f4d34b72640 5 monclient: authenticate success, global_id 4515 2024-03-29T11:27:47.513 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:47.520+0000 7f4d34b72640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:27:47.513 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:47.520+0000 7f4d2d59a640 1 -- 172.21.15.84:0/208326235 <== mon.0 v1:172.21.15.84:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4d240017d0 con 0x7f4d300b0860 2024-03-29T11:27:47.513 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:47.520+0000 7f4d2d59a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:27:47.513 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:47.520+0000 7f4d2d59a640 1 -- 172.21.15.84:0/208326235 <== mon.0 v1:172.21.15.84:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4236868694 0 0) 0x7f4d24001be0 con 0x7f4d300b0860 2024-03-29T11:27:47.513 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:47.520+0000 7f4d2d59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:27:47.513 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:47.520+0000 7f4d2d59a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:27:47.513 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:47.520+0000 7f4d2d59a640 10 monclient: dump: 2024-03-29T11:27:47.513 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T11:27:47.513 INFO:teuthology.misc.health.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:27:47.514 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T11:18:53.864025+0000 2024-03-29T11:27:47.514 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T11:18:53.864025+0000 2024-03-29T11:27:47.514 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T11:27:47.514 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:27:47.514 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T11:27:47.514 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T11:27:47.514 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T11:27:47.514 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:27:47.514 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:47.520+0000 7f4d34b72640 10 monclient: get_monmap_and_config success 2024-03-29T11:27:47.514 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:47.520+0000 7f4d34b72640 4 set_mon_vals no callback set 2024-03-29T11:27:47.514 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:47.521+0000 7f4d34b72640 10 monclient: shutdown 2024-03-29T11:27:47.514 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:47.521+0000 7f4d34b72640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:27:47.514 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:47.521+0000 7f4d34b72640 1 -- 172.21.15.84:0/208326235 >> v1:172.21.15.84:6789/0 conn(0x7f4d300b0860 legacy=0x7f4d3013e820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:47.514 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:47.521+0000 7f4d34b72640 1 -- 172.21.15.84:0/208326235 shutdown_connections 2024-03-29T11:27:47.515 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:47.521+0000 7f4d34b72640 1 -- 172.21.15.84:0/208326235 >> 172.21.15.84:0/208326235 conn(0x7f4d3006b660 msgr2=0x7f4d3006ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:27:47.515 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:47.521+0000 7f4d34b72640 1 -- 172.21.15.84:0/208326235 shutdown_connections 2024-03-29T11:27:47.515 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:47.521+0000 7f4d34b72640 1 -- 172.21.15.84:0/208326235 wait complete. 2024-03-29T11:27:47.515 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:47.522+0000 7f4d34b72640 10 monclient: build_initial_monmap 2024-03-29T11:27:47.515 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:47.522+0000 7f4d34b72640 10 monclient: monmap: 2024-03-29T11:27:47.515 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-29T11:27:47.515 INFO:teuthology.misc.health.smithi084.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:27:47.515 INFO:teuthology.misc.health.smithi084.stderr:last_changed 0.000000 2024-03-29T11:27:47.515 INFO:teuthology.misc.health.smithi084.stderr:created 0.000000 2024-03-29T11:27:47.515 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-29T11:27:47.516 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:27:47.516 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-29T11:27:47.516 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.noname-b 2024-03-29T11:27:47.516 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-c 2024-03-29T11:27:47.516 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:27:47.516 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:47.523+0000 7f4d34b72640 1 librados: starting msgr at 2024-03-29T11:27:47.516 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:47.523+0000 7f4d34b72640 1 librados: starting objecter 2024-03-29T11:27:47.516 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:47.523+0000 7f4d34b72640 1 Processor -- start 2024-03-29T11:27:47.516 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:47.523+0000 7f4d34b72640 1 -- start start 2024-03-29T11:27:47.516 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:47.523+0000 7f4d34b72640 1 librados: setting wanted keys 2024-03-29T11:27:47.516 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:47.523+0000 7f4d34b72640 1 librados: calling monclient init 2024-03-29T11:27:47.516 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:47.523+0000 7f4d34b72640 10 monclient: init 2024-03-29T11:27:47.516 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:47.524+0000 7f4d34b72640 10 monclient: _reopen_session rank -1 2024-03-29T11:27:47.516 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:47.524+0000 7f4d34b72640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:27:47.516 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:47.524+0000 7f4d34b72640 10 monclient(hunting): picked mon.noname-a con 0x7f4d300af8e0 addr v1:172.21.15.84:6789/0 2024-03-29T11:27:47.517 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:47.524+0000 7f4d34b72640 10 monclient(hunting): picked mon.noname-c con 0x7f4d300b0860 addr v1:172.21.15.84:6790/0 2024-03-29T11:27:47.517 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:47.524+0000 7f4d34b72640 10 monclient(hunting): picked mon.noname-b con 0x7f4d3013ef30 addr v1:172.21.15.179:6789/0 2024-03-29T11:27:47.517 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:47.524+0000 7f4d34b72640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4d3006f920 con 0x7f4d300af8e0 2024-03-29T11:27:47.517 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:47.524+0000 7f4d34b72640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4d30071b00 con 0x7f4d3013ef30 2024-03-29T11:27:47.517 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:47.524+0000 7f4d34b72640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4d30070d60 con 0x7f4d300b0860 2024-03-29T11:27:47.517 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:47.524+0000 7f4d34b72640 10 monclient(hunting): _renew_subs 2024-03-29T11:27:47.517 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:47.524+0000 7f4d34b72640 10 monclient(hunting): authenticate will time out at 1328.818229s 2024-03-29T11:27:47.522 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:47.527+0000 7f4d2ed9d640 1 --1- >> v1:172.21.15.179:6789/0 conn(0x7f4d3013ef30 0x7f4d3012d2b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.179:6789/0 says I am v1:172.21.15.84:46745/0 (socket says 172.21.15.84:46745) 2024-03-29T11:27:47.522 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:47.527+0000 7f4d2ed9d640 1 -- 172.21.15.84:0/302422827 learned_addr learned my addr 172.21.15.84:0/302422827 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-29T11:27:47.523 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:47.527+0000 7f4d1effd640 10 client.?.objecter ms_handle_connect 0x7f4d3013ef30 2024-03-29T11:27:47.523 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:47.527+0000 7f4d1effd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:27:47.523 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:47.527+0000 7f4d1effd640 1 -- 172.21.15.84:0/302422827 <== mon.1 v1:172.21.15.179:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2431602353 0 0) 0x7f4d30071b00 con 0x7f4d3013ef30 2024-03-29T11:27:47.523 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:47.527+0000 7f4d1effd640 10 monclient(hunting): _negotiate 2024-03-29T11:27:47.523 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:47.527+0000 7f4d1effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:27:47.523 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:47.527+0000 7f4d1effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:27:47.523 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:47.527+0000 7f4d1effd640 10 monclient(hunting): my global_id is 4468 2024-03-29T11:27:47.523 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:47.527+0000 7f4d1effd640 1 -- 172.21.15.84:0/302422827 --> v1:172.21.15.179:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4d10004eb0 con 0x7f4d3013ef30 2024-03-29T11:27:47.523 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:47.528+0000 7f4d1effd640 10 client.?.objecter ms_handle_connect 0x7f4d300b0860 2024-03-29T11:27:47.523 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:47.528+0000 7f4d1effd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:27:47.523 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:47.528+0000 7f4d1effd640 1 -- 172.21.15.84:0/302422827 <== mon.1 v1:172.21.15.179:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1371212912 0 0) 0x7f4d10004eb0 con 0x7f4d3013ef30 2024-03-29T11:27:47.523 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:47.528+0000 7f4d1effd640 1 -- 172.21.15.84:0/302422827 --> v1:172.21.15.179:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4d30071b00 con 0x7f4d3013ef30 2024-03-29T11:27:47.527 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:47.528+0000 7f4d1effd640 1 -- 172.21.15.84:0/302422827 <== mon.1 v1:172.21.15.179:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4236868694 0 0) 0x7f4d28004440 con 0x7f4d3013ef30 2024-03-29T11:27:47.527 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:47.528+0000 7f4d1effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:27:47.527 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:47.528+0000 7f4d1effd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:27:47.527 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:47.528+0000 7f4d1effd640 10 monclient(hunting): dump: 2024-03-29T11:27:47.527 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T11:27:47.527 INFO:teuthology.misc.health.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:27:47.527 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T11:18:53.864025+0000 2024-03-29T11:27:47.527 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T11:18:53.864025+0000 2024-03-29T11:27:47.527 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T11:27:47.527 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:27:47.527 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T11:27:47.528 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T11:27:47.528 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T11:27:47.528 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:27:47.528 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:47.528+0000 7f4d1effd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:27:47.528 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:47.528+0000 7f4d1effd640 1 -- 172.21.15.84:0/302422827 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3598714501 0 0) 0x7f4d30070d60 con 0x7f4d300b0860 2024-03-29T11:27:47.528 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:47.528+0000 7f4d1effd640 10 monclient(hunting): _negotiate 2024-03-29T11:27:47.528 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:47.528+0000 7f4d1effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:27:47.528 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:47.528+0000 7f4d1effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:27:47.528 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:47.528+0000 7f4d1effd640 10 monclient(hunting): my global_id is 4562 2024-03-29T11:27:47.528 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:47.528+0000 7f4d1effd640 1 -- 172.21.15.84:0/302422827 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4d10004eb0 con 0x7f4d300b0860 2024-03-29T11:27:47.528 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:47.529+0000 7f4d1effd640 1 -- 172.21.15.84:0/302422827 <== mon.2 v1:172.21.15.84:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2425839075 0 0) 0x7f4d10004eb0 con 0x7f4d300b0860 2024-03-29T11:27:47.528 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:47.529+0000 7f4d1effd640 1 -- 172.21.15.84:0/302422827 --> v1:172.21.15.84:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4d30070d60 con 0x7f4d300b0860 2024-03-29T11:27:47.529 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:47.529+0000 7f4d1effd640 1 -- 172.21.15.84:0/302422827 <== mon.2 v1:172.21.15.84:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4236868694 0 0) 0x7f4d20001750 con 0x7f4d300b0860 2024-03-29T11:27:47.529 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:47.529+0000 7f4d1effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:27:47.529 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:47.529+0000 7f4d1effd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:27:47.529 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:47.529+0000 7f4d1effd640 10 monclient(hunting): dump: 2024-03-29T11:27:47.529 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T11:27:47.529 INFO:teuthology.misc.health.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:27:47.529 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T11:18:53.864025+0000 2024-03-29T11:27:47.529 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T11:18:53.864025+0000 2024-03-29T11:27:47.529 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T11:27:47.529 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:27:47.529 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T11:27:47.529 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T11:27:47.529 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T11:27:47.529 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:27:47.529 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:47.529+0000 7f4d1effd640 1 -- 172.21.15.84:0/302422827 <== mon.2 v1:172.21.15.84:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2834571739 0 0) 0x7f4d30070d60 con 0x7f4d300b0860 2024-03-29T11:27:47.530 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:47.529+0000 7f4d1effd640 1 -- 172.21.15.84:0/302422827 >> v1:172.21.15.179:6789/0 conn(0x7f4d3013ef30 legacy=0x7f4d3012d2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:47.530 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:47.529+0000 7f4d1effd640 1 -- 172.21.15.84:0/302422827 >> v1:172.21.15.84:6789/0 conn(0x7f4d300af8e0 legacy=0x7f4d3006f210 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:27:47.530 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:47.530+0000 7f4d1effd640 10 monclient: _finish_hunting 0 2024-03-29T11:27:47.530 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:47.530+0000 7f4d1effd640 1 monclient: found mon.c 2024-03-29T11:27:47.530 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:47.530+0000 7f4d1effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:27:47.530 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:47.530+0000 7f4d1effd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-29T11:27:47.530 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:47.530+0000 7f4d1effd640 1 -- 172.21.15.84:0/302422827 --> v1:172.21.15.84:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d3006fe70 con 0x7f4d300b0860 2024-03-29T11:27:47.530 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:47.530+0000 7f4d1effd640 10 monclient: _finish_auth 0 2024-03-29T11:27:47.530 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:47.530+0000 7f4d1effd640 10 monclient: _check_auth_tickets 2024-03-29T11:27:47.530 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:47.530+0000 7f4d1effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:27:47.530 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:47.531+0000 7f4d1effd640 1 -- 172.21.15.84:0/302422827 <== mon.2 v1:172.21.15.84:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4d20001c40 con 0x7f4d300b0860 2024-03-29T11:27:47.530 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:47.531+0000 7f4d1effd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:27:47.530 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:47.531+0000 7f4d1effd640 1 -- 172.21.15.84:0/302422827 <== mon.2 v1:172.21.15.84:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4236868694 0 0) 0x7f4d20004980 con 0x7f4d300b0860 2024-03-29T11:27:47.531 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:47.531+0000 7f4d1effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:27:47.531 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:47.531+0000 7f4d1effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:27:47.531 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:47.531+0000 7f4d1effd640 10 monclient: dump: 2024-03-29T11:27:47.531 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T11:27:47.531 INFO:teuthology.misc.health.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:27:47.531 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T11:18:53.864025+0000 2024-03-29T11:27:47.531 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T11:18:53.864025+0000 2024-03-29T11:27:47.531 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T11:27:47.531 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:27:47.531 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T11:27:47.531 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T11:27:47.531 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T11:27:47.531 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:27:47.531 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:47.531+0000 7f4d1ffff640 4 set_mon_vals no callback set 2024-03-29T11:27:47.531 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:47.531+0000 7f4d34b72640 5 monclient: authenticate success, global_id 4562 2024-03-29T11:27:47.532 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:47.531+0000 7f4d34b72640 10 monclient: _renew_subs 2024-03-29T11:27:47.532 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:47.531+0000 7f4d34b72640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-29T11:27:47.532 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:47.531+0000 7f4d34b72640 1 -- 172.21.15.84:0/302422827 --> v1:172.21.15.84:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4d30071030 con 0x7f4d300b0860 2024-03-29T11:27:47.532 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:47.531+0000 7f4d34b72640 10 client.4562.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:27:47.532 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:47.531+0000 7f4d34b72640 10 monclient: _renew_subs 2024-03-29T11:27:47.532 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:47.531+0000 7f4d34b72640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-29T11:27:47.532 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:47.531+0000 7f4d34b72640 1 -- 172.21.15.84:0/302422827 --> v1:172.21.15.84:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f4d3006eae0 con 0x7f4d300b0860 2024-03-29T11:27:47.532 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:47.531+0000 7f4d34b72640 1 librados: init done 2024-03-29T11:27:47.533 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:47.534+0000 7f4d1effd640 1 -- 172.21.15.84:0/302422827 <== mon.2 v1:172.21.15.84:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4033942909 0 0) 0x7f4d2001a950 con 0x7f4d300b0860 2024-03-29T11:27:47.533 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:47.534+0000 7f4d1effd640 1 -- 172.21.15.84:0/302422827 <== mon.2 v1:172.21.15.84:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2808139587 0 0) 0x7f4d200848a0 con 0x7f4d300b0860 2024-03-29T11:27:47.533 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:47.534+0000 7f4d1effd640 10 client.4562.objecter ms_dispatch 0x7f4d30000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:27:47.533 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:47.534+0000 7f4d1effd640 3 client.4562.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:27:47.534 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:47.534+0000 7f4d1effd640 3 client.4562.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:27:47.534 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:47.535+0000 7f4d1effd640 20 client.4562.objecter dump_active .. 0 homeless 2024-03-29T11:27:47.534 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:47.535+0000 7f4d2e59c640 10 monclient: get_auth_request con 0x7f4d1006def0 auth_method 0 2024-03-29T11:27:47.534 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:47.535+0000 7f4d34b72640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:27:47.534 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:47.535+0000 7f4d34b72640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:27:47.534 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:47.535+0000 7f4d34b72640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-29T11:27:47.534 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:47.535+0000 7f4d34b72640 1 -- 172.21.15.84:0/302422827 --> v1:172.21.15.84:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4d30075f10 con 0x7f4d300b0860 2024-03-29T11:27:47.534 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:47.536+0000 7f4d1effd640 10 client.4562.objecter ms_handle_connect 0x7f4d1006def0 2024-03-29T11:27:47.535 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:47.546+0000 7f4d1effd640 1 -- 172.21.15.84:0/302422827 <== mon.2 v1:172.21.15.84:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f4d20050180 con 0x7f4d300b0860 2024-03-29T11:27:47.535 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:47.546+0000 7f4d1effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:27:47.535 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:47.546+0000 7f4d1effd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:27:47.865 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:47.876+0000 7f4d34b72640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:27:47.865 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:47.876+0000 7f4d34b72640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:27:47.865 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:47.876+0000 7f4d34b72640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-29T11:27:47.865 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:47.876+0000 7f4d34b72640 1 -- 172.21.15.84:0/302422827 --> v1:172.21.15.84:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4d30072330 con 0x7f4d300b0860 2024-03-29T11:27:47.866 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:47.877+0000 7f4d1effd640 1 -- 172.21.15.84:0/302422827 <== mon.2 v1:172.21.15.84:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4209477549) 0x7f4d200520b0 con 0x7f4d300b0860 2024-03-29T11:27:47.866 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:47.877+0000 7f4d1effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:27:47.866 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:47.877+0000 7f4d1effd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:27:47.866 INFO:teuthology.misc.health.smithi084.stdout: 2024-03-29T11:27:47.866 INFO:teuthology.misc.health.smithi084.stdout:{"epoch":10,"fsid":"2c3e002f-9eb0-4860-853b-4ef1ef562213","created":"2024-03-29T11:19:32.029169+0000","modified":"2024-03-29T11:19:34.200299+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:34.200299+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"5c39e276-2034-4716-bbc0-57a634eefa92","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"cd580e99-109f-4bf1-88c0-a5987bc8b9ed","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"11e60a5a-051d-453f-a190-66387cb35592","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"2c329fbe-bbc4-4030-bf87-d711cad04679","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f4213d4a-bd36-4433-91ae-d7e2b02481d3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"f0cf44ea-6055-48af-805b-aed858102ff9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"d895eaa0-cc0b-4ee4-9b5a-2dfcd89a60b7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"9bc5a7bc-8f52-4db9-96d6-de5e68d52eaa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:27:47.869 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:47.879+0000 7f4d34b72640 10 librados: watch_flush enter 2024-03-29T11:27:47.869 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:47.880+0000 7f4d34b72640 10 librados: watch_flush exit 2024-03-29T11:27:47.869 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:47.880+0000 7f4d34b72640 20 client.4562.objecter shutdown clearing up homeless session... 2024-03-29T11:27:47.869 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:47.880+0000 7f4d34b72640 10 client.4562.objecter successfully canceled tick 2024-03-29T11:27:47.869 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:47.880+0000 7f4d34b72640 1 -- 172.21.15.84:0/302422827 >> v1:172.21.15.179:6816/2981134980 conn(0x7f4d1006def0 legacy=0x7f4d100703b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:47.869 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:47.880+0000 7f4d34b72640 10 monclient: shutdown 2024-03-29T11:27:47.869 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:47.880+0000 7f4d34b72640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:27:47.869 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:47.880+0000 7f4d34b72640 1 -- 172.21.15.84:0/302422827 >> v1:172.21.15.84:6790/0 conn(0x7f4d300b0860 legacy=0x7f4d300713f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:47.869 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:47.880+0000 7f4d34b72640 1 -- 172.21.15.84:0/302422827 shutdown_connections 2024-03-29T11:27:47.869 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:47.880+0000 7f4d34b72640 1 -- 172.21.15.84:0/302422827 >> 172.21.15.84:0/302422827 conn(0x7f4d3006b660 msgr2=0x7f4d3013e5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:27:47.870 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:47.880+0000 7f4d34b72640 1 -- 172.21.15.84:0/302422827 shutdown_connections 2024-03-29T11:27:47.870 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:47.881+0000 7f4d34b72640 1 -- 172.21.15.84:0/302422827 wait complete. 2024-03-29T11:27:47.870 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:47.881+0000 7f4d34b72640 1 librados: shutdown 2024-03-29T11:27:47.887 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:27:54.694 DEBUG:teuthology.orchestra.run.smithi084:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:27:54.853 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:54.863+0000 7f8a5200f640 10 monclient: get_monmap_and_config 2024-03-29T11:27:54.853 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:54.863+0000 7f8a5200f640 10 monclient: build_initial_monmap 2024-03-29T11:27:54.853 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:54.863+0000 7f8a5200f640 10 monclient: monmap: 2024-03-29T11:27:54.853 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-29T11:27:54.853 INFO:teuthology.misc.health.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:27:54.853 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T11:27:54.864827+0000 2024-03-29T11:27:54.853 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T11:27:54.864827+0000 2024-03-29T11:27:54.853 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-29T11:27:54.853 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:27:54.853 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-29T11:27:54.853 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.noname-c 2024-03-29T11:27:54.854 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-b 2024-03-29T11:27:54.854 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:27:54.854 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:54.865+0000 7f8a5200f640 1 Processor -- start 2024-03-29T11:27:54.854 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:54.865+0000 7f8a5200f640 1 -- start start 2024-03-29T11:27:54.854 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:54.865+0000 7f8a5200f640 10 monclient: init 2024-03-29T11:27:54.864 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:54.866+0000 7f8a5200f640 10 monclient: _reopen_session rank -1 2024-03-29T11:27:54.864 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:54.866+0000 7f8a5200f640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:27:54.864 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:54.866+0000 7f8a5200f640 10 monclient(hunting): picked mon.noname-c con 0x7f8a4402b4c0 addr v1:172.21.15.179:6789/0 2024-03-29T11:27:54.865 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:54.866+0000 7f8a5200f640 10 monclient(hunting): picked mon.noname-b con 0x7f8a4400b8a0 addr v1:172.21.15.84:6790/0 2024-03-29T11:27:54.865 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:54.866+0000 7f8a5200f640 10 monclient(hunting): picked mon.noname-a con 0x7f8a4402a5a0 addr v1:172.21.15.84:6789/0 2024-03-29T11:27:54.865 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:54.866+0000 7f8a5200f640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8a4402c5c0 con 0x7f8a4402a5a0 2024-03-29T11:27:54.865 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:54.866+0000 7f8a5200f640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8a440c11a0 con 0x7f8a4402b4c0 2024-03-29T11:27:54.865 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:54.866+0000 7f8a5200f640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8a4402c830 con 0x7f8a4400b8a0 2024-03-29T11:27:54.865 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:54.866+0000 7f8a5200f640 10 monclient(hunting): _renew_subs 2024-03-29T11:27:54.865 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:54.866+0000 7f8a5200f640 10 monclient(hunting): authenticate will time out at 1336.160317s 2024-03-29T11:27:54.865 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:54.873+0000 7f8a4affd640 1 --1- >> v1:172.21.15.84:6790/0 conn(0x7f8a4400b8a0 0x7f8a440c0a90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.84:6790/0 says I am v1:172.21.15.84:51089/0 (socket says 172.21.15.84:51089) 2024-03-29T11:27:54.865 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:54.873+0000 7f8a4affd640 1 -- 172.21.15.84:0/366399770 learned_addr learned my addr 172.21.15.84:0/366399770 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-29T11:27:54.865 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:54.873+0000 7f8a4a7fc640 1 -- 172.21.15.84:0/366399770 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2751112613 0 0) 0x7f8a4402c830 con 0x7f8a4400b8a0 2024-03-29T11:27:54.865 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:54.873+0000 7f8a4a7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:27:54.865 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:54.873+0000 7f8a4a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:27:54.865 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:54.873+0000 7f8a4a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:27:54.865 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:54.873+0000 7f8a4a7fc640 10 monclient(hunting): my global_id is 4565 2024-03-29T11:27:54.866 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:54.873+0000 7f8a4a7fc640 1 -- 172.21.15.84:0/366399770 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8a3c003520 con 0x7f8a4400b8a0 2024-03-29T11:27:54.866 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:54.874+0000 7f8a4a7fc640 1 -- 172.21.15.84:0/366399770 <== mon.1 v1:172.21.15.179:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2060856041 0 0) 0x7f8a440c11a0 con 0x7f8a4402b4c0 2024-03-29T11:27:54.866 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:54.874+0000 7f8a4a7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:27:54.866 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:54.874+0000 7f8a4a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:27:54.866 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:54.874+0000 7f8a4a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:27:54.866 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:54.874+0000 7f8a4a7fc640 10 monclient(hunting): my global_id is 4471 2024-03-29T11:27:54.866 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:54.874+0000 7f8a4a7fc640 1 -- 172.21.15.84:0/366399770 --> v1:172.21.15.179:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8a4402c830 con 0x7f8a4402b4c0 2024-03-29T11:27:54.866 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:54.874+0000 7f8a4a7fc640 1 -- 172.21.15.84:0/366399770 <== mon.1 v1:172.21.15.179:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2308467136 0 0) 0x7f8a4402c830 con 0x7f8a4402b4c0 2024-03-29T11:27:54.867 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:54.874+0000 7f8a4a7fc640 1 -- 172.21.15.84:0/366399770 >> v1:172.21.15.84:6790/0 conn(0x7f8a4400b8a0 legacy=0x7f8a440c0a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:54.867 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:54.874+0000 7f8a4a7fc640 1 -- 172.21.15.84:0/366399770 >> v1:172.21.15.84:6789/0 conn(0x7f8a4402a5a0 legacy=0x7f8a440c6d40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:27:54.867 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:54.875+0000 7f8a4a7fc640 10 monclient: _finish_hunting 0 2024-03-29T11:27:54.868 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:54.875+0000 7f8a4a7fc640 1 monclient: found mon.noname-c 2024-03-29T11:27:54.868 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:54.875+0000 7f8a4a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:27:54.868 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:54.875+0000 7f8a4a7fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.179:6789/0 2024-03-29T11:27:54.868 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:54.875+0000 7f8a4a7fc640 1 -- 172.21.15.84:0/366399770 --> v1:172.21.15.179:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a4402afa0 con 0x7f8a4402b4c0 2024-03-29T11:27:54.868 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:54.875+0000 7f8a4a7fc640 10 monclient: _finish_auth 0 2024-03-29T11:27:54.868 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:54.875+0000 7f8a4a7fc640 10 monclient: _check_auth_tickets 2024-03-29T11:27:54.868 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:54.875+0000 7f8a4a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:27:54.868 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:54.875+0000 7f8a4a7fc640 1 -- 172.21.15.84:0/366399770 <== mon.1 v1:172.21.15.179:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4236868694 0 0) 0x7f8a40001670 con 0x7f8a4402b4c0 2024-03-29T11:27:54.868 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:54.875+0000 7f8a4a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:27:54.868 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:54.875+0000 7f8a4a7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:27:54.869 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:54.875+0000 7f8a4a7fc640 10 monclient: dump: 2024-03-29T11:27:54.869 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T11:27:54.869 INFO:teuthology.misc.health.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:27:54.869 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T11:18:53.864025+0000 2024-03-29T11:27:54.869 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T11:18:53.864025+0000 2024-03-29T11:27:54.869 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T11:27:54.869 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:27:54.869 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T11:27:54.869 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T11:27:54.869 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T11:27:54.869 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:27:54.869 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:54.877+0000 7f8a5200f640 5 monclient: authenticate success, global_id 4471 2024-03-29T11:27:54.869 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:54.877+0000 7f8a5200f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:27:54.869 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:54.878+0000 7f8a4a7fc640 1 -- 172.21.15.84:0/366399770 <== mon.1 v1:172.21.15.179:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8a400018d0 con 0x7f8a4402b4c0 2024-03-29T11:27:54.870 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:54.878+0000 7f8a4a7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:27:54.870 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:54.878+0000 7f8a5200f640 10 monclient: get_monmap_and_config success 2024-03-29T11:27:54.870 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:54.878+0000 7f8a5200f640 4 set_mon_vals no callback set 2024-03-29T11:27:54.870 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:54.879+0000 7f8a5200f640 10 monclient: shutdown 2024-03-29T11:27:54.870 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:54.879+0000 7f8a5200f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:27:54.870 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:54.879+0000 7f8a5200f640 1 -- 172.21.15.84:0/366399770 >> v1:172.21.15.179:6789/0 conn(0x7f8a4402b4c0 legacy=0x7f8a44006890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:54.870 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:54.879+0000 7f8a4a7fc640 1 -- 172.21.15.84:0/366399770 <== mon.1 v1:172.21.15.179:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4236868694 0 0) 0x7f8a40001ca0 con 0x7f8a4402b4c0 2024-03-29T11:27:54.870 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:54.879+0000 7f8a4a7fc640 10 monclient: discarding stray monitor message mon_map magic: 0 v1 2024-03-29T11:27:54.870 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:54.880+0000 7f8a5200f640 1 -- 172.21.15.84:0/366399770 shutdown_connections 2024-03-29T11:27:54.871 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:54.880+0000 7f8a5200f640 1 -- 172.21.15.84:0/366399770 >> 172.21.15.84:0/366399770 conn(0x7f8a440058f0 msgr2=0x7f8a44005d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:27:54.873 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:54.882+0000 7f8a5200f640 1 -- 172.21.15.84:0/366399770 shutdown_connections 2024-03-29T11:27:54.873 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:54.882+0000 7f8a5200f640 1 -- 172.21.15.84:0/366399770 wait complete. 2024-03-29T11:27:54.873 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:54.883+0000 7f8a5200f640 10 monclient: build_initial_monmap 2024-03-29T11:27:54.873 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:54.883+0000 7f8a5200f640 10 monclient: monmap: 2024-03-29T11:27:54.874 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-29T11:27:54.874 INFO:teuthology.misc.health.smithi084.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:27:54.874 INFO:teuthology.misc.health.smithi084.stderr:last_changed 0.000000 2024-03-29T11:27:54.874 INFO:teuthology.misc.health.smithi084.stderr:created 0.000000 2024-03-29T11:27:54.874 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-29T11:27:54.874 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:27:54.874 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-29T11:27:54.874 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.noname-b 2024-03-29T11:27:54.874 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-c 2024-03-29T11:27:54.874 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:27:54.874 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:54.883+0000 7f8a5200f640 1 librados: starting msgr at 2024-03-29T11:27:54.874 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:54.883+0000 7f8a5200f640 1 librados: starting objecter 2024-03-29T11:27:54.874 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:54.883+0000 7f8a5200f640 1 Processor -- start 2024-03-29T11:27:54.874 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:54.884+0000 7f8a5200f640 1 -- start start 2024-03-29T11:27:54.874 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:54.884+0000 7f8a5200f640 1 librados: setting wanted keys 2024-03-29T11:27:54.875 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:54.884+0000 7f8a5200f640 1 librados: calling monclient init 2024-03-29T11:27:54.875 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:54.884+0000 7f8a5200f640 10 monclient: init 2024-03-29T11:27:54.875 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:54.884+0000 7f8a5200f640 10 monclient: _reopen_session rank -1 2024-03-29T11:27:54.875 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:54.884+0000 7f8a5200f640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:27:54.875 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:54.884+0000 7f8a5200f640 10 monclient(hunting): picked mon.noname-b con 0x7f8a4400b8a0 addr v1:172.21.15.179:6789/0 2024-03-29T11:27:54.875 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:54.884+0000 7f8a5200f640 10 monclient(hunting): picked mon.noname-c con 0x7f8a4402a5a0 addr v1:172.21.15.84:6790/0 2024-03-29T11:27:54.875 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:54.884+0000 7f8a5200f640 10 monclient(hunting): picked mon.noname-a con 0x7f8a4402b4c0 addr v1:172.21.15.84:6789/0 2024-03-29T11:27:54.875 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:54.884+0000 7f8a5200f640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8a440c6130 con 0x7f8a4402b4c0 2024-03-29T11:27:54.875 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:54.884+0000 7f8a5200f640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8a44031100 con 0x7f8a4400b8a0 2024-03-29T11:27:54.875 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:54.884+0000 7f8a5200f640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8a44031870 con 0x7f8a4402a5a0 2024-03-29T11:27:54.875 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:54.884+0000 7f8a5200f640 10 monclient(hunting): _renew_subs 2024-03-29T11:27:54.875 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:54.884+0000 7f8a5200f640 10 monclient(hunting): authenticate will time out at 1336.178937s 2024-03-29T11:27:54.878 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:54.889+0000 7f8a4b7fe640 1 --1- >> v1:172.21.15.179:6789/0 conn(0x7f8a4400b8a0 0x7f8a4402eef0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.179:6789/0 says I am v1:172.21.15.84:52253/0 (socket says 172.21.15.84:52253) 2024-03-29T11:27:54.878 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:54.889+0000 7f8a4b7fe640 1 -- 172.21.15.84:0/3189033534 learned_addr learned my addr 172.21.15.84:0/3189033534 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-29T11:27:54.881 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:54.891+0000 7f8a37fff640 10 client.?.objecter ms_handle_connect 0x7f8a4402b4c0 2024-03-29T11:27:54.881 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:54.891+0000 7f8a37fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:27:54.882 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:54.891+0000 7f8a37fff640 10 client.?.objecter ms_handle_connect 0x7f8a4400b8a0 2024-03-29T11:27:54.882 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:54.891+0000 7f8a37fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:27:54.882 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:54.891+0000 7f8a37fff640 1 -- 172.21.15.84:0/3189033534 <== mon.0 v1:172.21.15.84:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1080939009 0 0) 0x7f8a440c6130 con 0x7f8a4402b4c0 2024-03-29T11:27:54.882 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:54.891+0000 7f8a37fff640 10 monclient(hunting): _negotiate 2024-03-29T11:27:54.882 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:54.891+0000 7f8a37fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:27:54.882 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:54.891+0000 7f8a37fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:27:54.882 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:54.891+0000 7f8a37fff640 10 monclient(hunting): my global_id is 4518 2024-03-29T11:27:54.882 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:54.891+0000 7f8a37fff640 1 -- 172.21.15.84:0/3189033534 --> v1:172.21.15.84:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8a30004eb0 con 0x7f8a4402b4c0 2024-03-29T11:27:54.882 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:54.892+0000 7f8a37fff640 1 -- 172.21.15.84:0/3189033534 <== mon.1 v1:172.21.15.179:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2002276689 0 0) 0x7f8a44031100 con 0x7f8a4400b8a0 2024-03-29T11:27:54.882 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:54.892+0000 7f8a37fff640 10 monclient(hunting): _negotiate 2024-03-29T11:27:54.882 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:54.892+0000 7f8a37fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:27:54.882 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:54.892+0000 7f8a37fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:27:54.882 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:54.892+0000 7f8a37fff640 10 monclient(hunting): my global_id is 4474 2024-03-29T11:27:54.882 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:54.892+0000 7f8a37fff640 1 -- 172.21.15.84:0/3189033534 --> v1:172.21.15.179:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8a440c6130 con 0x7f8a4400b8a0 2024-03-29T11:27:54.883 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:54.892+0000 7f8a37fff640 1 -- 172.21.15.84:0/3189033534 <== mon.0 v1:172.21.15.84:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2512436951 0 0) 0x7f8a30004eb0 con 0x7f8a4402b4c0 2024-03-29T11:27:54.883 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:54.892+0000 7f8a37fff640 1 -- 172.21.15.84:0/3189033534 --> v1:172.21.15.84:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8a44031100 con 0x7f8a4402b4c0 2024-03-29T11:27:54.883 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:54.892+0000 7f8a37fff640 1 -- 172.21.15.84:0/3189033534 <== mon.0 v1:172.21.15.84:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4236868694 0 0) 0x7f8a380017f0 con 0x7f8a4402b4c0 2024-03-29T11:27:54.883 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:54.892+0000 7f8a37fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:27:54.883 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:54.892+0000 7f8a37fff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:27:54.883 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:54.892+0000 7f8a37fff640 10 monclient(hunting): dump: 2024-03-29T11:27:54.883 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T11:27:54.883 INFO:teuthology.misc.health.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:27:54.883 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T11:18:53.864025+0000 2024-03-29T11:27:54.883 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T11:18:53.864025+0000 2024-03-29T11:27:54.884 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T11:27:54.884 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:27:54.884 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T11:27:54.884 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T11:27:54.884 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T11:27:54.884 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:27:54.884 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:54.892+0000 7f8a37fff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:27:54.884 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:54.892+0000 7f8a37fff640 10 client.?.objecter ms_handle_connect 0x7f8a4402a5a0 2024-03-29T11:27:54.884 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:54.892+0000 7f8a37fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:27:54.884 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:54.892+0000 7f8a37fff640 1 -- 172.21.15.84:0/3189033534 <== mon.1 v1:172.21.15.179:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 621146026 0 0) 0x7f8a440c6130 con 0x7f8a4400b8a0 2024-03-29T11:27:54.884 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:54.892+0000 7f8a37fff640 1 -- 172.21.15.84:0/3189033534 --> v1:172.21.15.179:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8a30004eb0 con 0x7f8a4400b8a0 2024-03-29T11:27:54.884 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:54.892+0000 7f8a37fff640 1 -- 172.21.15.84:0/3189033534 <== mon.1 v1:172.21.15.179:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4236868694 0 0) 0x7f8a40001ca0 con 0x7f8a4400b8a0 2024-03-29T11:27:54.884 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:54.892+0000 7f8a37fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:27:54.884 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:54.892+0000 7f8a37fff640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:27:54.884 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:54.892+0000 7f8a37fff640 10 monclient(hunting): dump: 2024-03-29T11:27:54.884 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T11:27:54.885 INFO:teuthology.misc.health.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:27:54.885 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T11:18:53.864025+0000 2024-03-29T11:27:54.885 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T11:18:53.864025+0000 2024-03-29T11:27:54.885 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T11:27:54.885 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:27:54.885 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T11:27:54.885 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T11:27:54.885 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T11:27:54.885 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:27:54.885 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:54.892+0000 7f8a37fff640 1 -- 172.21.15.84:0/3189033534 <== mon.0 v1:172.21.15.84:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2603641122 0 0) 0x7f8a44031100 con 0x7f8a4402b4c0 2024-03-29T11:27:54.885 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:54.892+0000 7f8a37fff640 1 -- 172.21.15.84:0/3189033534 >> v1:172.21.15.84:6790/0 conn(0x7f8a4402a5a0 legacy=0x7f8a4402f600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:54.885 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:54.892+0000 7f8a37fff640 1 -- 172.21.15.84:0/3189033534 >> v1:172.21.15.179:6789/0 conn(0x7f8a4400b8a0 legacy=0x7f8a4402eef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:54.885 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:54.892+0000 7f8a37fff640 10 monclient: _finish_hunting 0 2024-03-29T11:27:54.885 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:54.892+0000 7f8a37fff640 1 monclient: found mon.a 2024-03-29T11:27:54.886 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:54.893+0000 7f8a37fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:27:54.886 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:54.893+0000 7f8a37fff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.84:6789/0 2024-03-29T11:27:54.886 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:54.893+0000 7f8a37fff640 1 -- 172.21.15.84:0/3189033534 --> v1:172.21.15.84:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a440313e0 con 0x7f8a4402b4c0 2024-03-29T11:27:54.886 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:54.893+0000 7f8a37fff640 10 monclient: _finish_auth 0 2024-03-29T11:27:54.886 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:54.893+0000 7f8a37fff640 10 monclient: _check_auth_tickets 2024-03-29T11:27:54.886 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:54.893+0000 7f8a37fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:27:54.886 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:54.893+0000 7f8a5200f640 5 monclient: authenticate success, global_id 4518 2024-03-29T11:27:54.886 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:54.893+0000 7f8a5200f640 10 monclient: _renew_subs 2024-03-29T11:27:54.886 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:54.893+0000 7f8a5200f640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.84:6789/0 2024-03-29T11:27:54.887 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:54.893+0000 7f8a5200f640 1 -- 172.21.15.84:0/3189033534 --> v1:172.21.15.84:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8a44031b10 con 0x7f8a4402b4c0 2024-03-29T11:27:54.887 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:54.893+0000 7f8a5200f640 10 client.4518.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:27:54.887 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:54.893+0000 7f8a5200f640 10 monclient: _renew_subs 2024-03-29T11:27:54.887 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:54.893+0000 7f8a5200f640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.84:6789/0 2024-03-29T11:27:54.887 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:54.893+0000 7f8a5200f640 1 -- 172.21.15.84:0/3189033534 --> v1:172.21.15.84:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f8a440c0790 con 0x7f8a4402b4c0 2024-03-29T11:27:54.887 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:54.893+0000 7f8a5200f640 1 librados: init done 2024-03-29T11:27:54.889 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:54.895+0000 7f8a5200f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:27:54.889 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:54.895+0000 7f8a5200f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:27:54.889 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:54.895+0000 7f8a5200f640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.84:6789/0 2024-03-29T11:27:54.889 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:54.895+0000 7f8a5200f640 1 -- 172.21.15.84:0/3189033534 --> v1:172.21.15.84:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8a440d2380 con 0x7f8a4402b4c0 2024-03-29T11:27:54.889 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:54.896+0000 7f8a37fff640 1 -- 172.21.15.84:0/3189033534 <== mon.0 v1:172.21.15.84:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8a38001950 con 0x7f8a4402b4c0 2024-03-29T11:27:54.889 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:54.897+0000 7f8a37fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:27:54.889 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:54.897+0000 7f8a37fff640 1 -- 172.21.15.84:0/3189033534 <== mon.0 v1:172.21.15.84:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4236868694 0 0) 0x7f8a38003790 con 0x7f8a4402b4c0 2024-03-29T11:27:54.889 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:54.897+0000 7f8a37fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:27:54.889 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:54.897+0000 7f8a37fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:27:54.890 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:54.897+0000 7f8a37fff640 10 monclient: dump: 2024-03-29T11:27:54.890 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T11:27:54.890 INFO:teuthology.misc.health.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:27:54.890 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T11:18:53.864025+0000 2024-03-29T11:27:54.890 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T11:18:53.864025+0000 2024-03-29T11:27:54.890 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T11:27:54.890 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:27:54.890 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T11:27:54.890 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T11:27:54.890 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T11:27:54.890 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:27:54.890 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:54.897+0000 7f8a497fa640 4 set_mon_vals no callback set 2024-03-29T11:27:54.892 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:54.904+0000 7f8a37fff640 1 -- 172.21.15.84:0/3189033534 <== mon.0 v1:172.21.15.84:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4033942909 0 0) 0x7f8a380038f0 con 0x7f8a4402b4c0 2024-03-29T11:27:54.894 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:54.905+0000 7f8a4b7fe640 10 monclient: get_auth_request con 0x7f8a3006de90 auth_method 0 2024-03-29T11:27:54.894 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:54.906+0000 7f8a37fff640 1 -- 172.21.15.84:0/3189033534 <== mon.0 v1:172.21.15.84:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2808139587 0 0) 0x7f8a38083710 con 0x7f8a4402b4c0 2024-03-29T11:27:54.895 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:54.906+0000 7f8a37fff640 10 client.4518.objecter ms_dispatch 0x7f8a4c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:27:54.895 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:54.906+0000 7f8a37fff640 3 client.4518.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:27:54.895 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:54.906+0000 7f8a37fff640 3 client.4518.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:27:54.895 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:54.906+0000 7f8a37fff640 20 client.4518.objecter dump_active .. 0 homeless 2024-03-29T11:27:54.895 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:54.906+0000 7f8a37fff640 10 client.4518.objecter ms_handle_connect 0x7f8a3006de90 2024-03-29T11:27:54.900 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:54.909+0000 7f8a37fff640 1 -- 172.21.15.84:0/3189033534 <== mon.0 v1:172.21.15.84:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f8a3804ee20 con 0x7f8a4402b4c0 2024-03-29T11:27:54.900 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:54.909+0000 7f8a37fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:27:54.900 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:54.909+0000 7f8a37fff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:27:55.236 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:55.247+0000 7f8a5200f640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:27:55.236 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:55.247+0000 7f8a5200f640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:27:55.236 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:55.247+0000 7f8a5200f640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.84:6789/0 2024-03-29T11:27:55.236 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:55.247+0000 7f8a5200f640 1 -- 172.21.15.84:0/3189033534 --> v1:172.21.15.84:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8a44031ca0 con 0x7f8a4402b4c0 2024-03-29T11:27:55.241 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:55.252+0000 7f8a37fff640 1 -- 172.21.15.84:0/3189033534 <== mon.0 v1:172.21.15.84:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4209477549) 0x7f8a38050f20 con 0x7f8a4402b4c0 2024-03-29T11:27:55.241 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:55.252+0000 7f8a37fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:27:55.241 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:55.252+0000 7f8a37fff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:27:55.245 INFO:teuthology.misc.health.smithi084.stdout: 2024-03-29T11:27:55.245 INFO:teuthology.misc.health.smithi084.stdout:{"epoch":10,"fsid":"2c3e002f-9eb0-4860-853b-4ef1ef562213","created":"2024-03-29T11:19:32.029169+0000","modified":"2024-03-29T11:19:34.200299+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:34.200299+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"5c39e276-2034-4716-bbc0-57a634eefa92","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"cd580e99-109f-4bf1-88c0-a5987bc8b9ed","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"11e60a5a-051d-453f-a190-66387cb35592","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"2c329fbe-bbc4-4030-bf87-d711cad04679","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f4213d4a-bd36-4433-91ae-d7e2b02481d3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"f0cf44ea-6055-48af-805b-aed858102ff9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"d895eaa0-cc0b-4ee4-9b5a-2dfcd89a60b7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"9bc5a7bc-8f52-4db9-96d6-de5e68d52eaa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:27:55.249 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:55.259+0000 7f8a5200f640 10 librados: watch_flush enter 2024-03-29T11:27:55.249 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:55.259+0000 7f8a5200f640 10 librados: watch_flush exit 2024-03-29T11:27:55.249 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:55.259+0000 7f8a5200f640 20 client.4518.objecter shutdown clearing up homeless session... 2024-03-29T11:27:55.249 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:55.259+0000 7f8a5200f640 10 client.4518.objecter successfully canceled tick 2024-03-29T11:27:55.249 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:55.259+0000 7f8a5200f640 1 -- 172.21.15.84:0/3189033534 >> v1:172.21.15.179:6816/2981134980 conn(0x7f8a3006de90 legacy=0x7f8a30070350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:55.249 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:55.259+0000 7f8a5200f640 10 monclient: shutdown 2024-03-29T11:27:55.249 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:55.259+0000 7f8a5200f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:27:55.250 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:55.259+0000 7f8a5200f640 1 -- 172.21.15.84:0/3189033534 >> v1:172.21.15.84:6789/0 conn(0x7f8a4402b4c0 legacy=0x7f8a441897d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:55.250 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:55.260+0000 7f8a5200f640 1 -- 172.21.15.84:0/3189033534 shutdown_connections 2024-03-29T11:27:55.250 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:55.260+0000 7f8a5200f640 1 -- 172.21.15.84:0/3189033534 >> 172.21.15.84:0/3189033534 conn(0x7f8a440058f0 msgr2=0x7f8a440d35c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:27:55.253 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:55.264+0000 7f8a5200f640 1 -- 172.21.15.84:0/3189033534 shutdown_connections 2024-03-29T11:27:55.253 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:55.264+0000 7f8a5200f640 1 -- 172.21.15.84:0/3189033534 wait complete. 2024-03-29T11:27:55.253 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:27:55.264+0000 7f8a5200f640 1 librados: shutdown 2024-03-29T11:27:55.270 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:28:02.079 DEBUG:teuthology.orchestra.run.smithi084:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:28:02.235 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:02.246+0000 7f36dec2d640 10 monclient: get_monmap_and_config 2024-03-29T11:28:02.235 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:02.246+0000 7f36dec2d640 10 monclient: build_initial_monmap 2024-03-29T11:28:02.236 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:02.246+0000 7f36dec2d640 10 monclient: monmap: 2024-03-29T11:28:02.236 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-29T11:28:02.236 INFO:teuthology.misc.health.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:28:02.236 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T11:28:02.247451+0000 2024-03-29T11:28:02.236 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T11:28:02.247451+0000 2024-03-29T11:28:02.236 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-29T11:28:02.236 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:28:02.236 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-29T11:28:02.236 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.noname-c 2024-03-29T11:28:02.236 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-b 2024-03-29T11:28:02.236 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:28:02.237 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:02.247+0000 7f36dec2d640 1 Processor -- start 2024-03-29T11:28:02.237 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:02.248+0000 7f36dec2d640 1 -- start start 2024-03-29T11:28:02.237 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:02.248+0000 7f36dec2d640 10 monclient: init 2024-03-29T11:28:02.238 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:02.248+0000 7f36dec2d640 10 monclient: _reopen_session rank -1 2024-03-29T11:28:02.238 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:02.248+0000 7f36dec2d640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:28:02.238 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:02.248+0000 7f36dec2d640 10 monclient(hunting): picked mon.noname-c con 0x7f36d00389e0 addr v1:172.21.15.179:6789/0 2024-03-29T11:28:02.238 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:02.249+0000 7f36dec2d640 10 monclient(hunting): picked mon.noname-b con 0x7f36d0038dc0 addr v1:172.21.15.84:6790/0 2024-03-29T11:28:02.238 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:02.249+0000 7f36dec2d640 10 monclient(hunting): picked mon.noname-a con 0x7f36d0030660 addr v1:172.21.15.84:6789/0 2024-03-29T11:28:02.238 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:02.249+0000 7f36dec2d640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f36d00c54c0 con 0x7f36d0030660 2024-03-29T11:28:02.238 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:02.249+0000 7f36dec2d640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f36d0036430 con 0x7f36d00389e0 2024-03-29T11:28:02.238 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:02.249+0000 7f36dec2d640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f36d0039840 con 0x7f36d0038dc0 2024-03-29T11:28:02.238 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:02.249+0000 7f36dec2d640 10 monclient(hunting): _renew_subs 2024-03-29T11:28:02.239 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:02.249+0000 7f36dec2d640 10 monclient(hunting): authenticate will time out at 1343.543163s 2024-03-29T11:28:02.239 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:02.249+0000 7f36de42c640 1 --1- >> v1:172.21.15.84:6789/0 conn(0x7f36d0030660 0x7f36d00d3170 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.84:6789/0 says I am v1:172.21.15.84:54386/0 (socket says 172.21.15.84:54386) 2024-03-29T11:28:02.239 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:02.249+0000 7f36de42c640 1 -- 172.21.15.84:0/2244076818 learned_addr learned my addr 172.21.15.84:0/2244076818 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-29T11:28:02.240 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:02.251+0000 7f36dcc29640 1 -- 172.21.15.84:0/2244076818 <== mon.0 v1:172.21.15.84:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2628369620 0 0) 0x7f36d00c54c0 con 0x7f36d0030660 2024-03-29T11:28:02.240 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:02.251+0000 7f36dcc29640 10 monclient(hunting): _negotiate 2024-03-29T11:28:02.241 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:02.251+0000 7f36dcc29640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:28:02.241 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:02.251+0000 7f36dcc29640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:28:02.241 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:02.251+0000 7f36dcc29640 10 monclient(hunting): my global_id is 4521 2024-03-29T11:28:02.241 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:02.251+0000 7f36dcc29640 1 -- 172.21.15.84:0/2244076818 --> v1:172.21.15.84:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f36c4003520 con 0x7f36d0030660 2024-03-29T11:28:02.242 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:02.252+0000 7f36dcc29640 1 -- 172.21.15.84:0/2244076818 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2588192414 0 0) 0x7f36d0039840 con 0x7f36d0038dc0 2024-03-29T11:28:02.242 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:02.252+0000 7f36dcc29640 10 monclient(hunting): _negotiate 2024-03-29T11:28:02.242 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:02.252+0000 7f36dcc29640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:28:02.242 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:02.252+0000 7f36dcc29640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:28:02.242 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:02.252+0000 7f36dcc29640 10 monclient(hunting): my global_id is 4571 2024-03-29T11:28:02.242 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:02.253+0000 7f36dcc29640 1 -- 172.21.15.84:0/2244076818 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f36d00c54c0 con 0x7f36d0038dc0 2024-03-29T11:28:02.243 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:02.253+0000 7f36dcc29640 1 -- 172.21.15.84:0/2244076818 <== mon.0 v1:172.21.15.84:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3984674548 0 0) 0x7f36c4003520 con 0x7f36d0030660 2024-03-29T11:28:02.243 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:02.253+0000 7f36dcc29640 1 -- 172.21.15.84:0/2244076818 >> v1:172.21.15.84:6790/0 conn(0x7f36d0038dc0 legacy=0x7f36d00cadd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:02.244 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:02.253+0000 7f36dcc29640 1 -- 172.21.15.84:0/2244076818 >> v1:172.21.15.179:6789/0 conn(0x7f36d00389e0 legacy=0x7f36d0033520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:02.244 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:02.253+0000 7f36dcc29640 10 monclient: _finish_hunting 0 2024-03-29T11:28:02.244 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:02.253+0000 7f36dcc29640 1 monclient: found mon.noname-a 2024-03-29T11:28:02.244 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:02.253+0000 7f36dcc29640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:28:02.244 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:02.253+0000 7f36dcc29640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.84:6789/0 2024-03-29T11:28:02.244 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:02.254+0000 7f36dcc29640 1 -- 172.21.15.84:0/2244076818 --> v1:172.21.15.84:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36d0035690 con 0x7f36d0030660 2024-03-29T11:28:02.244 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:02.254+0000 7f36dcc29640 10 monclient: _finish_auth 0 2024-03-29T11:28:02.244 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:02.254+0000 7f36dcc29640 10 monclient: _check_auth_tickets 2024-03-29T11:28:02.244 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:02.254+0000 7f36dcc29640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:28:02.244 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:02.254+0000 7f36dcc29640 1 -- 172.21.15.84:0/2244076818 <== mon.0 v1:172.21.15.84:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4236868694 0 0) 0x7f36c8001690 con 0x7f36d0030660 2024-03-29T11:28:02.244 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:02.254+0000 7f36dcc29640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:28:02.244 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:02.254+0000 7f36dcc29640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:28:02.244 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:02.254+0000 7f36dcc29640 10 monclient: dump: 2024-03-29T11:28:02.245 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T11:28:02.245 INFO:teuthology.misc.health.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:28:02.245 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T11:18:53.864025+0000 2024-03-29T11:28:02.245 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T11:18:53.864025+0000 2024-03-29T11:28:02.245 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T11:28:02.245 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:28:02.245 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T11:28:02.245 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T11:28:02.245 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T11:28:02.245 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:28:02.245 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:02.254+0000 7f36dec2d640 5 monclient: authenticate success, global_id 4521 2024-03-29T11:28:02.245 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:02.254+0000 7f36dec2d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:28:02.246 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:02.255+0000 7f36dcc29640 1 -- 172.21.15.84:0/2244076818 <== mon.0 v1:172.21.15.84:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f36c80017f0 con 0x7f36d0030660 2024-03-29T11:28:02.246 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:02.255+0000 7f36dcc29640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:28:02.246 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:02.255+0000 7f36dec2d640 10 monclient: get_monmap_and_config success 2024-03-29T11:28:02.247 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:02.255+0000 7f36dec2d640 4 set_mon_vals no callback set 2024-03-29T11:28:02.247 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:02.256+0000 7f36dec2d640 10 monclient: shutdown 2024-03-29T11:28:02.247 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:02.256+0000 7f36dec2d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:28:02.247 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:02.256+0000 7f36dec2d640 1 -- 172.21.15.84:0/2244076818 >> v1:172.21.15.84:6789/0 conn(0x7f36d0030660 legacy=0x7f36d00d3170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:02.247 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:02.257+0000 7f36dec2d640 1 -- 172.21.15.84:0/2244076818 shutdown_connections 2024-03-29T11:28:02.247 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:02.257+0000 7f36dec2d640 1 -- 172.21.15.84:0/2244076818 >> 172.21.15.84:0/2244076818 conn(0x7f36d0005c00 msgr2=0x7f36d0006010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:28:02.247 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:02.257+0000 7f36dec2d640 1 -- 172.21.15.84:0/2244076818 shutdown_connections 2024-03-29T11:28:02.248 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:02.257+0000 7f36dec2d640 1 -- 172.21.15.84:0/2244076818 wait complete. 2024-03-29T11:28:02.248 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:02.258+0000 7f36dec2d640 10 monclient: build_initial_monmap 2024-03-29T11:28:02.248 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:02.258+0000 7f36dec2d640 10 monclient: monmap: 2024-03-29T11:28:02.249 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-29T11:28:02.249 INFO:teuthology.misc.health.smithi084.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:28:02.249 INFO:teuthology.misc.health.smithi084.stderr:last_changed 0.000000 2024-03-29T11:28:02.249 INFO:teuthology.misc.health.smithi084.stderr:created 0.000000 2024-03-29T11:28:02.249 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-29T11:28:02.249 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:28:02.249 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-29T11:28:02.249 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.noname-b 2024-03-29T11:28:02.249 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-c 2024-03-29T11:28:02.249 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:28:02.249 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:02.259+0000 7f36dec2d640 1 librados: starting msgr at 2024-03-29T11:28:02.249 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:02.259+0000 7f36dec2d640 1 librados: starting objecter 2024-03-29T11:28:02.250 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:02.259+0000 7f36dec2d640 1 Processor -- start 2024-03-29T11:28:02.250 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:02.259+0000 7f36dec2d640 1 -- start start 2024-03-29T11:28:02.250 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:02.259+0000 7f36dec2d640 1 librados: setting wanted keys 2024-03-29T11:28:02.250 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:02.259+0000 7f36dec2d640 1 librados: calling monclient init 2024-03-29T11:28:02.250 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:02.259+0000 7f36dec2d640 10 monclient: init 2024-03-29T11:28:02.250 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:02.260+0000 7f36dec2d640 10 monclient: _reopen_session rank -1 2024-03-29T11:28:02.250 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:02.260+0000 7f36dec2d640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:28:02.250 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:02.260+0000 7f36dec2d640 10 monclient(hunting): picked mon.noname-b con 0x7f36d0030660 addr v1:172.21.15.179:6789/0 2024-03-29T11:28:02.251 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:02.260+0000 7f36dec2d640 10 monclient(hunting): picked mon.noname-a con 0x7f36d00389e0 addr v1:172.21.15.84:6789/0 2024-03-29T11:28:02.252 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:02.263+0000 7f36dec2d640 10 monclient(hunting): picked mon.noname-c con 0x7f36d00c0670 addr v1:172.21.15.84:6790/0 2024-03-29T11:28:02.252 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:02.263+0000 7f36dec2d640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f36d00c0ab0 con 0x7f36d00389e0 2024-03-29T11:28:02.252 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:02.263+0000 7f36dec2d640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f36d00c5650 con 0x7f36d0030660 2024-03-29T11:28:02.252 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:02.263+0000 7f36dec2d640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f36d00c58f0 con 0x7f36d00c0670 2024-03-29T11:28:02.252 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:02.263+0000 7f36dec2d640 10 monclient(hunting): _renew_subs 2024-03-29T11:28:02.258 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:02.263+0000 7f36dec2d640 10 monclient(hunting): authenticate will time out at 1343.557456s 2024-03-29T11:28:02.259 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:02.263+0000 7f36de42c640 1 --1- >> v1:172.21.15.84:6790/0 conn(0x7f36d00c0670 0x7f36d01645d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.84:6790/0 says I am v1:172.21.15.84:55899/0 (socket says 172.21.15.84:55899) 2024-03-29T11:28:02.259 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:02.263+0000 7f36de42c640 1 -- 172.21.15.84:0/3926628696 learned_addr learned my addr 172.21.15.84:0/3926628696 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-29T11:28:02.259 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:02.263+0000 7f36ce7fc640 10 client.?.objecter ms_handle_connect 0x7f36d00c0670 2024-03-29T11:28:02.259 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:02.263+0000 7f36ce7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:28:02.259 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:02.263+0000 7f36ce7fc640 1 -- 172.21.15.84:0/3926628696 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4259826956 0 0) 0x7f36d00c58f0 con 0x7f36d00c0670 2024-03-29T11:28:02.259 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:02.263+0000 7f36ce7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:28:02.259 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:02.263+0000 7f36ce7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:28:02.259 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:02.263+0000 7f36ce7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:28:02.259 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:02.263+0000 7f36ce7fc640 10 monclient(hunting): my global_id is 4574 2024-03-29T11:28:02.259 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:02.263+0000 7f36ce7fc640 1 -- 172.21.15.84:0/3926628696 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f36bc004eb0 con 0x7f36d00c0670 2024-03-29T11:28:02.259 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:02.264+0000 7f36ce7fc640 1 -- 172.21.15.84:0/3926628696 <== mon.2 v1:172.21.15.84:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2310099418 0 0) 0x7f36bc004eb0 con 0x7f36d00c0670 2024-03-29T11:28:02.259 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:02.264+0000 7f36ce7fc640 1 -- 172.21.15.84:0/3926628696 --> v1:172.21.15.84:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f36d00c58f0 con 0x7f36d00c0670 2024-03-29T11:28:02.259 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:02.264+0000 7f36ce7fc640 1 -- 172.21.15.84:0/3926628696 <== mon.2 v1:172.21.15.84:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4236868694 0 0) 0x7f36c8004940 con 0x7f36d00c0670 2024-03-29T11:28:02.260 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:02.264+0000 7f36ce7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:28:02.260 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:02.264+0000 7f36ce7fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:28:02.260 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:02.264+0000 7f36ce7fc640 10 monclient(hunting): dump: 2024-03-29T11:28:02.260 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T11:28:02.260 INFO:teuthology.misc.health.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:28:02.260 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T11:18:53.864025+0000 2024-03-29T11:28:02.260 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T11:18:53.864025+0000 2024-03-29T11:28:02.260 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T11:28:02.260 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:28:02.260 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T11:28:02.260 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T11:28:02.260 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T11:28:02.261 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:28:02.261 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:02.264+0000 7f36ce7fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:28:02.261 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:02.264+0000 7f36ce7fc640 10 client.?.objecter ms_handle_connect 0x7f36d00389e0 2024-03-29T11:28:02.261 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:02.264+0000 7f36ce7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:28:02.261 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:02.264+0000 7f36ce7fc640 1 -- 172.21.15.84:0/3926628696 <== mon.2 v1:172.21.15.84:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 618540332 0 0) 0x7f36d00c58f0 con 0x7f36d00c0670 2024-03-29T11:28:02.261 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:02.264+0000 7f36ce7fc640 1 -- 172.21.15.84:0/3926628696 >> v1:172.21.15.179:6789/0 conn(0x7f36d0030660 legacy=0x7f36d002f9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:02.261 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:02.264+0000 7f36ce7fc640 1 -- 172.21.15.84:0/3926628696 >> v1:172.21.15.84:6789/0 conn(0x7f36d00389e0 legacy=0x7f36d00bff60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:02.261 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:02.264+0000 7f36ce7fc640 10 monclient: _finish_hunting 0 2024-03-29T11:28:02.261 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:02.264+0000 7f36ce7fc640 1 monclient: found mon.c 2024-03-29T11:28:02.261 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:02.264+0000 7f36ce7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:28:02.261 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:02.264+0000 7f36ce7fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-29T11:28:02.261 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:02.264+0000 7f36ce7fc640 1 -- 172.21.15.84:0/3926628696 --> v1:172.21.15.84:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36d00c73e0 con 0x7f36d00c0670 2024-03-29T11:28:02.261 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:02.264+0000 7f36ce7fc640 10 monclient: _finish_auth 0 2024-03-29T11:28:02.261 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:02.264+0000 7f36ce7fc640 10 monclient: _check_auth_tickets 2024-03-29T11:28:02.261 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:02.264+0000 7f36ce7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:28:02.262 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:02.264+0000 7f36dec2d640 5 monclient: authenticate success, global_id 4574 2024-03-29T11:28:02.262 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:02.264+0000 7f36dec2d640 10 monclient: _renew_subs 2024-03-29T11:28:02.262 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:02.264+0000 7f36dec2d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-29T11:28:02.262 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:02.264+0000 7f36dec2d640 1 -- 172.21.15.84:0/3926628696 --> v1:172.21.15.84:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f36d00c0cb0 con 0x7f36d00c0670 2024-03-29T11:28:02.262 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:02.264+0000 7f36dec2d640 10 client.4574.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:28:02.262 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:02.264+0000 7f36dec2d640 10 monclient: _renew_subs 2024-03-29T11:28:02.262 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:02.264+0000 7f36dec2d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-29T11:28:02.262 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:02.264+0000 7f36dec2d640 1 -- 172.21.15.84:0/3926628696 --> v1:172.21.15.84:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f36d0164ec0 con 0x7f36d00c0670 2024-03-29T11:28:02.262 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:02.264+0000 7f36dec2d640 1 librados: init done 2024-03-29T11:28:02.262 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:02.266+0000 7f36ce7fc640 1 -- 172.21.15.84:0/3926628696 <== mon.2 v1:172.21.15.84:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f36c8001aa0 con 0x7f36d00c0670 2024-03-29T11:28:02.262 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:02.266+0000 7f36ce7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:28:02.262 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:02.266+0000 7f36ce7fc640 1 -- 172.21.15.84:0/3926628696 <== mon.2 v1:172.21.15.84:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4236868694 0 0) 0x7f36c8004f70 con 0x7f36d00c0670 2024-03-29T11:28:02.262 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:02.266+0000 7f36ce7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:28:02.262 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:02.266+0000 7f36ce7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:28:02.263 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:02.266+0000 7f36ce7fc640 10 monclient: dump: 2024-03-29T11:28:02.263 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T11:28:02.263 INFO:teuthology.misc.health.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:28:02.263 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T11:18:53.864025+0000 2024-03-29T11:28:02.263 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T11:18:53.864025+0000 2024-03-29T11:28:02.263 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T11:28:02.263 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:28:02.263 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T11:28:02.263 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T11:28:02.263 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T11:28:02.263 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:28:02.263 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:02.266+0000 7f36c3fff640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:28:02.263 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:02.266+0000 7f36c3fff640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:28:02.263 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:02.266+0000 7f36c3fff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-29T11:28:02.263 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:02.266+0000 7f36c3fff640 1 -- 172.21.15.84:0/3926628696 --> v1:172.21.15.84:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f36d0007b90 con 0x7f36d00c0670 2024-03-29T11:28:02.264 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:02.266+0000 7f36cffff640 4 set_mon_vals no callback set 2024-03-29T11:28:02.264 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:02.267+0000 7f36ce7fc640 1 -- 172.21.15.84:0/3926628696 <== mon.2 v1:172.21.15.84:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4033942909 0 0) 0x7f36c8005190 con 0x7f36d00c0670 2024-03-29T11:28:02.264 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:02.267+0000 7f36ce7fc640 1 -- 172.21.15.84:0/3926628696 <== mon.2 v1:172.21.15.84:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2808139587 0 0) 0x7f36c8084c90 con 0x7f36d00c0670 2024-03-29T11:28:02.264 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:02.267+0000 7f36ce7fc640 10 client.4574.objecter ms_dispatch 0x7f36d8000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:28:02.264 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:02.267+0000 7f36ce7fc640 3 client.4574.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:28:02.264 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:02.267+0000 7f36ce7fc640 3 client.4574.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:28:02.264 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:02.267+0000 7f36ce7fc640 20 client.4574.objecter dump_active .. 0 homeless 2024-03-29T11:28:02.265 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:02.267+0000 7f36ddc2b640 10 monclient: get_auth_request con 0x7f36bc06dcb0 auth_method 0 2024-03-29T11:28:02.265 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:02.268+0000 7f36ce7fc640 10 client.4574.objecter ms_handle_connect 0x7f36bc06dcb0 2024-03-29T11:28:02.265 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:02.275+0000 7f36ce7fc640 1 -- 172.21.15.84:0/3926628696 <== mon.2 v1:172.21.15.84:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f36c8050570 con 0x7f36d00c0670 2024-03-29T11:28:02.265 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:02.275+0000 7f36ce7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:28:02.265 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:02.275+0000 7f36ce7fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:28:02.592 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:02.603+0000 7f36c3fff640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:28:02.592 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:02.603+0000 7f36c3fff640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:28:02.592 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:02.603+0000 7f36c3fff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-29T11:28:02.593 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:02.603+0000 7f36c3fff640 1 -- 172.21.15.84:0/3926628696 --> v1:172.21.15.84:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f36d01651c0 con 0x7f36d00c0670 2024-03-29T11:28:02.593 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:02.604+0000 7f36ce7fc640 1 -- 172.21.15.84:0/3926628696 <== mon.2 v1:172.21.15.84:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4209477549) 0x7f36d01651c0 con 0x7f36d00c0670 2024-03-29T11:28:02.593 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:02.604+0000 7f36ce7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:28:02.593 INFO:teuthology.misc.health.smithi084.stdout: 2024-03-29T11:28:02.594 INFO:teuthology.misc.health.smithi084.stdout:{"epoch":10,"fsid":"2c3e002f-9eb0-4860-853b-4ef1ef562213","created":"2024-03-29T11:19:32.029169+0000","modified":"2024-03-29T11:19:34.200299+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:34.200299+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"5c39e276-2034-4716-bbc0-57a634eefa92","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"cd580e99-109f-4bf1-88c0-a5987bc8b9ed","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"11e60a5a-051d-453f-a190-66387cb35592","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"2c329fbe-bbc4-4030-bf87-d711cad04679","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f4213d4a-bd36-4433-91ae-d7e2b02481d3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"f0cf44ea-6055-48af-805b-aed858102ff9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"d895eaa0-cc0b-4ee4-9b5a-2dfcd89a60b7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"9bc5a7bc-8f52-4db9-96d6-de5e68d52eaa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:28:02.594 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:02.604+0000 7f36ce7fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:28:02.598 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:02.607+0000 7f36c3fff640 10 librados: watch_flush enter 2024-03-29T11:28:02.598 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:02.608+0000 7f36c3fff640 10 librados: watch_flush exit 2024-03-29T11:28:02.598 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:02.608+0000 7f36c3fff640 20 client.4574.objecter shutdown clearing up homeless session... 2024-03-29T11:28:02.598 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:02.608+0000 7f36c3fff640 10 client.4574.objecter successfully canceled tick 2024-03-29T11:28:02.599 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:02.608+0000 7f36c3fff640 1 -- 172.21.15.84:0/3926628696 >> v1:172.21.15.179:6816/2981134980 conn(0x7f36bc06dcb0 legacy=0x7f36bc070170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:02.599 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:02.608+0000 7f36c3fff640 10 monclient: shutdown 2024-03-29T11:28:02.599 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:02.608+0000 7f36c3fff640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:28:02.599 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:02.608+0000 7f36c3fff640 1 -- 172.21.15.84:0/3926628696 >> v1:172.21.15.84:6790/0 conn(0x7f36d00c0670 legacy=0x7f36d01645d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:02.599 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:02.608+0000 7f36c3fff640 1 -- 172.21.15.84:0/3926628696 shutdown_connections 2024-03-29T11:28:02.599 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:02.608+0000 7f36c3fff640 1 -- 172.21.15.84:0/3926628696 >> 172.21.15.84:0/3926628696 conn(0x7f36d0005c00 msgr2=0x7f36d0034090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:28:02.599 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:02.608+0000 7f36c3fff640 1 -- 172.21.15.84:0/3926628696 shutdown_connections 2024-03-29T11:28:02.599 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:02.609+0000 7f36c3fff640 1 -- 172.21.15.84:0/3926628696 wait complete. 2024-03-29T11:28:02.599 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:02.609+0000 7f36c3fff640 1 librados: shutdown 2024-03-29T11:28:02.615 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:28:09.421 DEBUG:teuthology.orchestra.run.smithi084:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:28:09.556 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:09.567+0000 7f675504a640 10 monclient: get_monmap_and_config 2024-03-29T11:28:09.556 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:09.567+0000 7f675504a640 10 monclient: build_initial_monmap 2024-03-29T11:28:09.556 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:09.567+0000 7f675504a640 10 monclient: monmap: 2024-03-29T11:28:09.556 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-29T11:28:09.556 INFO:teuthology.misc.health.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:28:09.556 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T11:28:09.568149+0000 2024-03-29T11:28:09.557 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T11:28:09.568149+0000 2024-03-29T11:28:09.557 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-29T11:28:09.557 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:28:09.557 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-29T11:28:09.557 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.noname-c 2024-03-29T11:28:09.557 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-b 2024-03-29T11:28:09.557 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:28:09.558 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:09.568+0000 7f675504a640 1 Processor -- start 2024-03-29T11:28:09.558 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:09.568+0000 7f675504a640 1 -- start start 2024-03-29T11:28:09.558 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:09.568+0000 7f675504a640 10 monclient: init 2024-03-29T11:28:09.558 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:09.569+0000 7f675504a640 10 monclient: _reopen_session rank -1 2024-03-29T11:28:09.558 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:09.569+0000 7f675504a640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:28:09.558 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:09.569+0000 7f675504a640 10 monclient(hunting): picked mon.noname-a con 0x7f67501457f0 addr v1:172.21.15.84:6789/0 2024-03-29T11:28:09.558 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:09.569+0000 7f675504a640 10 monclient(hunting): picked mon.noname-c con 0x7f675013f650 addr v1:172.21.15.179:6789/0 2024-03-29T11:28:09.558 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:09.569+0000 7f675504a640 10 monclient(hunting): picked mon.noname-b con 0x7f67500a9d70 addr v1:172.21.15.84:6790/0 2024-03-29T11:28:09.559 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:09.569+0000 7f675504a640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f67500aaef0 con 0x7f67501457f0 2024-03-29T11:28:09.559 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:09.569+0000 7f675504a640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6750145bd0 con 0x7f675013f650 2024-03-29T11:28:09.559 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:09.569+0000 7f675504a640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6750145eb0 con 0x7f67500a9d70 2024-03-29T11:28:09.559 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:09.569+0000 7f675504a640 10 monclient(hunting): _renew_subs 2024-03-29T11:28:09.559 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:09.569+0000 7f675504a640 10 monclient(hunting): authenticate will time out at 1350.863445s 2024-03-29T11:28:09.559 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:09.569+0000 7f674ed9d640 1 --1- >> v1:172.21.15.84:6789/0 conn(0x7f67501457f0 0x7f675013ef40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.84:6789/0 says I am v1:172.21.15.84:53240/0 (socket says 172.21.15.84:53240) 2024-03-29T11:28:09.559 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:09.569+0000 7f674ed9d640 1 -- 172.21.15.84:0/2824860791 learned_addr learned my addr 172.21.15.84:0/2824860791 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-29T11:28:09.559 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:09.569+0000 7f674dd9b640 1 -- 172.21.15.84:0/2824860791 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1569684033 0 0) 0x7f6750145eb0 con 0x7f67500a9d70 2024-03-29T11:28:09.559 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:09.569+0000 7f674dd9b640 10 monclient(hunting): _negotiate 2024-03-29T11:28:09.559 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:09.569+0000 7f674dd9b640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:28:09.560 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:09.569+0000 7f674dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:28:09.561 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:09.569+0000 7f674dd9b640 10 monclient(hunting): my global_id is 4577 2024-03-29T11:28:09.562 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:09.569+0000 7f674dd9b640 1 -- 172.21.15.84:0/2824860791 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6738003520 con 0x7f67500a9d70 2024-03-29T11:28:09.562 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:09.570+0000 7f674dd9b640 1 -- 172.21.15.84:0/2824860791 <== mon.2 v1:172.21.15.84:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3478801496 0 0) 0x7f6738003520 con 0x7f67500a9d70 2024-03-29T11:28:09.562 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:09.570+0000 7f674dd9b640 1 -- 172.21.15.84:0/2824860791 >> v1:172.21.15.179:6789/0 conn(0x7f675013f650 legacy=0x7f67500b2620 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:28:09.566 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:09.572+0000 7f674dd9b640 1 -- 172.21.15.84:0/2824860791 >> v1:172.21.15.84:6789/0 conn(0x7f67501457f0 legacy=0x7f675013ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:09.566 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:09.572+0000 7f674dd9b640 10 monclient: _finish_hunting 0 2024-03-29T11:28:09.566 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:09.572+0000 7f674dd9b640 1 monclient: found mon.noname-b 2024-03-29T11:28:09.566 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:09.572+0000 7f674dd9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:28:09.566 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:09.572+0000 7f674dd9b640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.84:6790/0 2024-03-29T11:28:09.566 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:09.572+0000 7f674dd9b640 1 -- 172.21.15.84:0/2824860791 --> v1:172.21.15.84:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6750146340 con 0x7f67500a9d70 2024-03-29T11:28:09.566 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:09.572+0000 7f674dd9b640 10 monclient: _finish_auth 0 2024-03-29T11:28:09.566 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:09.572+0000 7f674dd9b640 10 monclient: _check_auth_tickets 2024-03-29T11:28:09.566 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:09.572+0000 7f674dd9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:28:09.567 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:09.572+0000 7f674dd9b640 1 -- 172.21.15.84:0/2824860791 <== mon.2 v1:172.21.15.84:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4236868694 0 0) 0x7f6740001690 con 0x7f67500a9d70 2024-03-29T11:28:09.567 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:09.572+0000 7f674dd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:28:09.567 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:09.572+0000 7f674dd9b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:28:09.567 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:09.572+0000 7f674dd9b640 10 monclient: dump: 2024-03-29T11:28:09.567 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T11:28:09.567 INFO:teuthology.misc.health.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:28:09.567 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T11:18:53.864025+0000 2024-03-29T11:28:09.567 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T11:18:53.864025+0000 2024-03-29T11:28:09.567 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T11:28:09.567 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:28:09.567 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T11:28:09.567 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T11:28:09.567 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T11:28:09.567 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:28:09.567 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:09.572+0000 7f675504a640 5 monclient: authenticate success, global_id 4577 2024-03-29T11:28:09.568 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:09.572+0000 7f675504a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:28:09.568 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:09.573+0000 7f674dd9b640 1 -- 172.21.15.84:0/2824860791 <== mon.2 v1:172.21.15.84:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6740001920 con 0x7f67500a9d70 2024-03-29T11:28:09.568 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:09.573+0000 7f674dd9b640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:28:09.568 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:09.573+0000 7f674dd9b640 1 -- 172.21.15.84:0/2824860791 <== mon.2 v1:172.21.15.84:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4236868694 0 0) 0x7f6740001cb0 con 0x7f67500a9d70 2024-03-29T11:28:09.568 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:09.573+0000 7f675504a640 10 monclient: get_monmap_and_config success 2024-03-29T11:28:09.568 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:09.573+0000 7f675504a640 4 set_mon_vals no callback set 2024-03-29T11:28:09.568 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:09.573+0000 7f674dd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:28:09.568 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:09.573+0000 7f674dd9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:28:09.568 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:09.573+0000 7f674dd9b640 10 monclient: dump: 2024-03-29T11:28:09.568 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T11:28:09.568 INFO:teuthology.misc.health.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:28:09.568 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T11:18:53.864025+0000 2024-03-29T11:28:09.568 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T11:18:53.864025+0000 2024-03-29T11:28:09.568 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T11:28:09.568 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:28:09.569 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T11:28:09.569 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T11:28:09.569 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T11:28:09.569 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:28:09.569 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:09.573+0000 7f675504a640 10 monclient: shutdown 2024-03-29T11:28:09.569 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:09.573+0000 7f675504a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:28:09.569 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:09.573+0000 7f675504a640 1 -- 172.21.15.84:0/2824860791 >> v1:172.21.15.84:6790/0 conn(0x7f67500a9d70 legacy=0x7f675014d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:09.569 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:09.574+0000 7f675504a640 1 -- 172.21.15.84:0/2824860791 shutdown_connections 2024-03-29T11:28:09.569 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:09.574+0000 7f675504a640 1 -- 172.21.15.84:0/2824860791 >> 172.21.15.84:0/2824860791 conn(0x7f6750066a00 msgr2=0x7f6750066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:28:09.569 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:09.575+0000 7f675504a640 1 -- 172.21.15.84:0/2824860791 shutdown_connections 2024-03-29T11:28:09.569 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:09.575+0000 7f675504a640 1 -- 172.21.15.84:0/2824860791 wait complete. 2024-03-29T11:28:09.569 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:09.576+0000 7f675504a640 10 monclient: build_initial_monmap 2024-03-29T11:28:09.569 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:09.576+0000 7f675504a640 10 monclient: monmap: 2024-03-29T11:28:09.569 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-29T11:28:09.569 INFO:teuthology.misc.health.smithi084.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:28:09.570 INFO:teuthology.misc.health.smithi084.stderr:last_changed 0.000000 2024-03-29T11:28:09.570 INFO:teuthology.misc.health.smithi084.stderr:created 0.000000 2024-03-29T11:28:09.570 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-29T11:28:09.570 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:28:09.570 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-29T11:28:09.570 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.noname-b 2024-03-29T11:28:09.570 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-c 2024-03-29T11:28:09.570 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:28:09.570 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:09.577+0000 7f675504a640 1 librados: starting msgr at 2024-03-29T11:28:09.570 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:09.577+0000 7f675504a640 1 librados: starting objecter 2024-03-29T11:28:09.574 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:09.577+0000 7f675504a640 1 Processor -- start 2024-03-29T11:28:09.575 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:09.577+0000 7f675504a640 1 -- start start 2024-03-29T11:28:09.575 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:09.577+0000 7f675504a640 1 librados: setting wanted keys 2024-03-29T11:28:09.575 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:09.577+0000 7f675504a640 1 librados: calling monclient init 2024-03-29T11:28:09.575 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:09.577+0000 7f675504a640 10 monclient: init 2024-03-29T11:28:09.575 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:09.578+0000 7f675504a640 10 monclient: _reopen_session rank -1 2024-03-29T11:28:09.575 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:09.578+0000 7f675504a640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:28:09.575 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:09.579+0000 7f675504a640 10 monclient(hunting): picked mon.noname-b con 0x7f67500a9d70 addr v1:172.21.15.179:6789/0 2024-03-29T11:28:09.575 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:09.579+0000 7f675504a640 10 monclient(hunting): picked mon.noname-c con 0x7f675013f650 addr v1:172.21.15.84:6790/0 2024-03-29T11:28:09.575 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:09.579+0000 7f675504a640 10 monclient(hunting): picked mon.noname-a con 0x7f67501457f0 addr v1:172.21.15.84:6789/0 2024-03-29T11:28:09.575 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:09.579+0000 7f675504a640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f67500aeee0 con 0x7f67501457f0 2024-03-29T11:28:09.575 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:09.579+0000 7f675504a640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f67500affd0 con 0x7f67500a9d70 2024-03-29T11:28:09.575 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:09.579+0000 7f675504a640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f67500b02b0 con 0x7f675013f650 2024-03-29T11:28:09.575 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:09.579+0000 7f675504a640 10 monclient(hunting): _renew_subs 2024-03-29T11:28:09.576 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:09.579+0000 7f675504a640 10 monclient(hunting): authenticate will time out at 1350.873763s 2024-03-29T11:28:09.576 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:09.579+0000 7f674f59e640 1 --1- >> v1:172.21.15.84:6789/0 conn(0x7f67501457f0 0x7f67501e19f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.84:6789/0 says I am v1:172.21.15.84:55002/0 (socket says 172.21.15.84:55002) 2024-03-29T11:28:09.576 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:09.579+0000 7f674f59e640 1 -- 172.21.15.84:0/2351127047 learned_addr learned my addr 172.21.15.84:0/2351127047 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-29T11:28:09.576 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:09.580+0000 7f67377fe640 10 client.?.objecter ms_handle_connect 0x7f67501457f0 2024-03-29T11:28:09.576 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:09.580+0000 7f67377fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:28:09.576 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:09.580+0000 7f67377fe640 10 client.?.objecter ms_handle_connect 0x7f675013f650 2024-03-29T11:28:09.576 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:09.580+0000 7f67377fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:28:09.576 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:09.580+0000 7f67377fe640 1 -- 172.21.15.84:0/2351127047 <== mon.0 v1:172.21.15.84:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2780698736 0 0) 0x7f67500aeee0 con 0x7f67501457f0 2024-03-29T11:28:09.576 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:09.580+0000 7f67377fe640 10 monclient(hunting): _negotiate 2024-03-29T11:28:09.576 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:09.580+0000 7f67377fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:28:09.576 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:09.580+0000 7f67377fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:28:09.576 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:09.580+0000 7f67377fe640 10 monclient(hunting): my global_id is 4530 2024-03-29T11:28:09.576 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:09.580+0000 7f67377fe640 1 -- 172.21.15.84:0/2351127047 --> v1:172.21.15.84:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f673c004eb0 con 0x7f67501457f0 2024-03-29T11:28:09.576 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:09.580+0000 7f67377fe640 1 -- 172.21.15.84:0/2351127047 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1924525850 0 0) 0x7f67500b02b0 con 0x7f675013f650 2024-03-29T11:28:09.577 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:09.580+0000 7f67377fe640 10 monclient(hunting): _negotiate 2024-03-29T11:28:09.577 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:09.580+0000 7f67377fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:28:09.577 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:09.580+0000 7f67377fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:28:09.577 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:09.580+0000 7f67377fe640 10 monclient(hunting): my global_id is 4580 2024-03-29T11:28:09.577 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:09.580+0000 7f67377fe640 1 -- 172.21.15.84:0/2351127047 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f67500aeee0 con 0x7f675013f650 2024-03-29T11:28:09.577 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:09.580+0000 7f67377fe640 1 -- 172.21.15.84:0/2351127047 <== mon.0 v1:172.21.15.84:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1532098726 0 0) 0x7f673c004eb0 con 0x7f67501457f0 2024-03-29T11:28:09.577 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:09.580+0000 7f67377fe640 1 -- 172.21.15.84:0/2351127047 --> v1:172.21.15.84:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f67500b02b0 con 0x7f67501457f0 2024-03-29T11:28:09.577 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:09.580+0000 7f67377fe640 1 -- 172.21.15.84:0/2351127047 <== mon.0 v1:172.21.15.84:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4236868694 0 0) 0x7f6740001610 con 0x7f67501457f0 2024-03-29T11:28:09.577 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:09.580+0000 7f67377fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:28:09.577 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:09.580+0000 7f67377fe640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:28:09.577 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:09.580+0000 7f67377fe640 10 monclient(hunting): dump: 2024-03-29T11:28:09.577 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T11:28:09.577 INFO:teuthology.misc.health.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:28:09.577 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T11:18:53.864025+0000 2024-03-29T11:28:09.577 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T11:18:53.864025+0000 2024-03-29T11:28:09.578 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T11:28:09.578 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:28:09.578 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T11:28:09.578 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T11:28:09.578 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T11:28:09.578 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:28:09.578 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:09.580+0000 7f67377fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:28:09.578 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:09.581+0000 7f67377fe640 1 -- 172.21.15.84:0/2351127047 <== mon.0 v1:172.21.15.84:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 60065782 0 0) 0x7f67500b02b0 con 0x7f67501457f0 2024-03-29T11:28:09.578 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:09.581+0000 7f67377fe640 1 -- 172.21.15.84:0/2351127047 >> v1:172.21.15.84:6790/0 conn(0x7f675013f650 legacy=0x7f67500b0de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:09.579 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:09.581+0000 7f67377fe640 1 -- 172.21.15.84:0/2351127047 >> v1:172.21.15.179:6789/0 conn(0x7f67500a9d70 legacy=0x7f67500b06d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:28:09.579 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:09.583+0000 7f67377fe640 10 monclient: _finish_hunting 0 2024-03-29T11:28:09.579 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:09.583+0000 7f67377fe640 1 monclient: found mon.a 2024-03-29T11:28:09.579 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:09.583+0000 7f67377fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:28:09.579 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:09.583+0000 7f67377fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.84:6789/0 2024-03-29T11:28:09.579 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:09.583+0000 7f67377fe640 1 -- 172.21.15.84:0/2351127047 --> v1:172.21.15.84:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67500aea40 con 0x7f67501457f0 2024-03-29T11:28:09.579 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:09.583+0000 7f67377fe640 10 monclient: _finish_auth 0 2024-03-29T11:28:09.579 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:09.583+0000 7f67377fe640 10 monclient: _check_auth_tickets 2024-03-29T11:28:09.579 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:09.583+0000 7f67377fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:28:09.579 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:09.583+0000 7f67377fe640 10 client.?.objecter ms_handle_connect 0x7f67500a9d70 2024-03-29T11:28:09.579 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:09.583+0000 7f67377fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:28:09.579 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:09.583+0000 7f675504a640 5 monclient: authenticate success, global_id 4530 2024-03-29T11:28:09.579 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:09.583+0000 7f675504a640 10 monclient: _renew_subs 2024-03-29T11:28:09.579 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:09.583+0000 7f675504a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.84:6789/0 2024-03-29T11:28:09.579 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:09.583+0000 7f675504a640 1 -- 172.21.15.84:0/2351127047 --> v1:172.21.15.84:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f67500b1680 con 0x7f67501457f0 2024-03-29T11:28:09.580 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:09.583+0000 7f67377fe640 1 -- 172.21.15.84:0/2351127047 <== mon.0 v1:172.21.15.84:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6740003960 con 0x7f67501457f0 2024-03-29T11:28:09.580 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:09.583+0000 7f67377fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:28:09.580 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:09.583+0000 7f675504a640 10 client.4530.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:28:09.580 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:09.583+0000 7f67377fe640 1 -- 172.21.15.84:0/2351127047 <== mon.0 v1:172.21.15.84:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4236868694 0 0) 0x7f6740003d70 con 0x7f67501457f0 2024-03-29T11:28:09.580 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:09.583+0000 7f675504a640 10 monclient: _renew_subs 2024-03-29T11:28:09.580 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:09.583+0000 7f675504a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.84:6789/0 2024-03-29T11:28:09.580 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:09.583+0000 7f675504a640 1 -- 172.21.15.84:0/2351127047 --> v1:172.21.15.84:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f67500ae030 con 0x7f67501457f0 2024-03-29T11:28:09.580 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:09.583+0000 7f67377fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:28:09.580 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:09.583+0000 7f67377fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:28:09.580 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:09.583+0000 7f674d59a640 4 set_mon_vals no callback set 2024-03-29T11:28:09.580 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:09.583+0000 7f67377fe640 10 monclient: dump: 2024-03-29T11:28:09.580 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T11:28:09.580 INFO:teuthology.misc.health.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:28:09.580 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T11:18:53.864025+0000 2024-03-29T11:28:09.581 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T11:18:53.864025+0000 2024-03-29T11:28:09.581 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T11:28:09.581 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:28:09.581 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T11:28:09.581 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T11:28:09.581 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T11:28:09.581 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:28:09.581 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:09.583+0000 7f675504a640 1 librados: init done 2024-03-29T11:28:09.581 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:09.584+0000 7f675504a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:28:09.581 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:09.584+0000 7f675504a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:28:09.581 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:09.584+0000 7f675504a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.84:6789/0 2024-03-29T11:28:09.581 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:09.584+0000 7f675504a640 1 -- 172.21.15.84:0/2351127047 --> v1:172.21.15.84:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f67500af090 con 0x7f67501457f0 2024-03-29T11:28:09.582 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:09.586+0000 7f67377fe640 1 -- 172.21.15.84:0/2351127047 <== mon.0 v1:172.21.15.84:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4033942909 0 0) 0x7f6740003ed0 con 0x7f67501457f0 2024-03-29T11:28:09.583 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:09.587+0000 7f67377fe640 1 -- 172.21.15.84:0/2351127047 <== mon.0 v1:172.21.15.84:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2808139587 0 0) 0x7f6740084900 con 0x7f67501457f0 2024-03-29T11:28:09.583 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:09.587+0000 7f67377fe640 10 client.4530.objecter ms_dispatch 0x7f6750000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:28:09.583 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:09.587+0000 7f67377fe640 3 client.4530.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:28:09.583 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:09.587+0000 7f67377fe640 3 client.4530.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:28:09.583 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:09.587+0000 7f67377fe640 20 client.4530.objecter dump_active .. 0 homeless 2024-03-29T11:28:09.583 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:09.587+0000 7f674ed9d640 10 monclient: get_auth_request con 0x7f673c06dba0 auth_method 0 2024-03-29T11:28:09.583 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:09.587+0000 7f67377fe640 10 client.4530.objecter ms_handle_connect 0x7f673c06dba0 2024-03-29T11:28:09.583 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:09.593+0000 7f67377fe640 1 -- 172.21.15.84:0/2351127047 <== mon.0 v1:172.21.15.84:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f67400501e0 con 0x7f67501457f0 2024-03-29T11:28:09.583 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:09.593+0000 7f67377fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:28:09.583 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:09.593+0000 7f67377fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:28:09.903 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:09.913+0000 7f675504a640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:28:09.903 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:09.913+0000 7f675504a640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:28:09.903 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:09.913+0000 7f675504a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.84:6789/0 2024-03-29T11:28:09.903 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:09.913+0000 7f675504a640 1 -- 172.21.15.84:0/2351127047 --> v1:172.21.15.84:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f67500af090 con 0x7f67501457f0 2024-03-29T11:28:09.904 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:09.914+0000 7f67377fe640 1 -- 172.21.15.84:0/2351127047 <== mon.0 v1:172.21.15.84:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4209477549) 0x7f67500af090 con 0x7f67501457f0 2024-03-29T11:28:09.904 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:09.914+0000 7f67377fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:28:09.904 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:09.914+0000 7f67377fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:28:09.907 INFO:teuthology.misc.health.smithi084.stdout: 2024-03-29T11:28:09.907 INFO:teuthology.misc.health.smithi084.stdout:{"epoch":10,"fsid":"2c3e002f-9eb0-4860-853b-4ef1ef562213","created":"2024-03-29T11:19:32.029169+0000","modified":"2024-03-29T11:19:34.200299+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:34.200299+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"5c39e276-2034-4716-bbc0-57a634eefa92","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"cd580e99-109f-4bf1-88c0-a5987bc8b9ed","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"11e60a5a-051d-453f-a190-66387cb35592","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"2c329fbe-bbc4-4030-bf87-d711cad04679","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f4213d4a-bd36-4433-91ae-d7e2b02481d3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"f0cf44ea-6055-48af-805b-aed858102ff9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"d895eaa0-cc0b-4ee4-9b5a-2dfcd89a60b7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"9bc5a7bc-8f52-4db9-96d6-de5e68d52eaa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:28:09.911 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:09.921+0000 7f675504a640 10 librados: watch_flush enter 2024-03-29T11:28:09.911 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:09.921+0000 7f675504a640 10 librados: watch_flush exit 2024-03-29T11:28:09.911 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:09.921+0000 7f675504a640 20 client.4530.objecter shutdown clearing up homeless session... 2024-03-29T11:28:09.911 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:09.921+0000 7f675504a640 10 client.4530.objecter successfully canceled tick 2024-03-29T11:28:09.912 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:09.922+0000 7f675504a640 1 -- 172.21.15.84:0/2351127047 >> v1:172.21.15.179:6816/2981134980 conn(0x7f673c06dba0 legacy=0x7f673c070060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:09.912 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:09.922+0000 7f675504a640 10 monclient: shutdown 2024-03-29T11:28:09.912 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:09.922+0000 7f675504a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:28:09.912 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:09.922+0000 7f675504a640 1 -- 172.21.15.84:0/2351127047 >> v1:172.21.15.84:6789/0 conn(0x7f67501457f0 legacy=0x7f67501e19f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:09.912 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:09.922+0000 7f675504a640 1 -- 172.21.15.84:0/2351127047 shutdown_connections 2024-03-29T11:28:09.912 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:09.922+0000 7f675504a640 1 -- 172.21.15.84:0/2351127047 >> 172.21.15.84:0/2351127047 conn(0x7f6750066a00 msgr2=0x7f67500b5810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:28:09.912 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:09.923+0000 7f675504a640 1 -- 172.21.15.84:0/2351127047 shutdown_connections 2024-03-29T11:28:09.913 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:09.923+0000 7f675504a640 1 -- 172.21.15.84:0/2351127047 wait complete. 2024-03-29T11:28:09.913 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:09.923+0000 7f675504a640 1 librados: shutdown 2024-03-29T11:28:09.930 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:28:16.736 DEBUG:teuthology.orchestra.run.smithi084:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:28:16.888 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:16.898+0000 7f3197169640 10 monclient: get_monmap_and_config 2024-03-29T11:28:16.889 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:16.898+0000 7f3197169640 10 monclient: build_initial_monmap 2024-03-29T11:28:16.889 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:16.898+0000 7f3197169640 10 monclient: monmap: 2024-03-29T11:28:16.889 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-29T11:28:16.889 INFO:teuthology.misc.health.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:28:16.889 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T11:28:16.900411+0000 2024-03-29T11:28:16.889 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T11:28:16.900411+0000 2024-03-29T11:28:16.889 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-29T11:28:16.889 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:28:16.889 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-29T11:28:16.889 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.noname-c 2024-03-29T11:28:16.890 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-b 2024-03-29T11:28:16.890 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:28:16.891 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:16.899+0000 7f3197169640 1 Processor -- start 2024-03-29T11:28:16.891 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:16.899+0000 7f3197169640 1 -- start start 2024-03-29T11:28:16.891 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:16.899+0000 7f3197169640 10 monclient: init 2024-03-29T11:28:16.891 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:16.900+0000 7f3197169640 10 monclient: _reopen_session rank -1 2024-03-29T11:28:16.891 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:16.900+0000 7f3197169640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:28:16.891 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:16.900+0000 7f3197169640 10 monclient(hunting): picked mon.noname-c con 0x7f3188006940 addr v1:172.21.15.179:6789/0 2024-03-29T11:28:16.891 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:16.900+0000 7f3197169640 10 monclient(hunting): picked mon.noname-b con 0x7f3188035480 addr v1:172.21.15.84:6790/0 2024-03-29T11:28:16.891 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:16.900+0000 7f3197169640 10 monclient(hunting): picked mon.noname-a con 0x7f31880c4ec0 addr v1:172.21.15.84:6789/0 2024-03-29T11:28:16.891 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:16.900+0000 7f3197169640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3188036f30 con 0x7f31880c4ec0 2024-03-29T11:28:16.892 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:16.900+0000 7f3197169640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3188037210 con 0x7f3188006940 2024-03-29T11:28:16.892 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:16.900+0000 7f3197169640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3188034f60 con 0x7f3188035480 2024-03-29T11:28:16.892 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:16.900+0000 7f3197169640 10 monclient(hunting): _renew_subs 2024-03-29T11:28:16.892 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:16.900+0000 7f3197169640 10 monclient(hunting): authenticate will time out at 1358.195874s 2024-03-29T11:28:16.892 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:16.901+0000 7f3196167640 1 --1- >> v1:172.21.15.179:6789/0 conn(0x7f3188006940 0x7f31880cad00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.179:6789/0 says I am v1:172.21.15.84:45436/0 (socket says 172.21.15.84:45436) 2024-03-29T11:28:16.892 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:16.901+0000 7f3196167640 1 -- 172.21.15.84:0/2977951920 learned_addr learned my addr 172.21.15.84:0/2977951920 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-29T11:28:16.895 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:16.901+0000 7f3195165640 1 -- 172.21.15.84:0/2977951920 <== mon.1 v1:172.21.15.179:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1347796481 0 0) 0x7f3188037210 con 0x7f3188006940 2024-03-29T11:28:16.895 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:16.901+0000 7f3195165640 10 monclient(hunting): _negotiate 2024-03-29T11:28:16.895 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:16.901+0000 7f3195165640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:28:16.895 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:16.901+0000 7f3195165640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:28:16.895 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:16.901+0000 7f3195165640 10 monclient(hunting): my global_id is 4483 2024-03-29T11:28:16.896 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:16.901+0000 7f3195165640 1 -- 172.21.15.84:0/2977951920 --> v1:172.21.15.179:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f317c003520 con 0x7f3188006940 2024-03-29T11:28:16.896 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:16.901+0000 7f3195165640 1 -- 172.21.15.84:0/2977951920 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 134457962 0 0) 0x7f3188034f60 con 0x7f3188035480 2024-03-29T11:28:16.896 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:16.901+0000 7f3195165640 10 monclient(hunting): _negotiate 2024-03-29T11:28:16.896 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:16.901+0000 7f3195165640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:28:16.896 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:16.901+0000 7f3195165640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:28:16.896 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:16.901+0000 7f3195165640 10 monclient(hunting): my global_id is 4583 2024-03-29T11:28:16.896 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:16.901+0000 7f3195165640 1 -- 172.21.15.84:0/2977951920 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3188037210 con 0x7f3188035480 2024-03-29T11:28:16.896 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:16.902+0000 7f3195165640 1 -- 172.21.15.84:0/2977951920 <== mon.1 v1:172.21.15.179:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2419757823 0 0) 0x7f317c003520 con 0x7f3188006940 2024-03-29T11:28:16.896 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:16.902+0000 7f3195165640 1 -- 172.21.15.84:0/2977951920 >> v1:172.21.15.84:6790/0 conn(0x7f3188035480 legacy=0x7f31880c47b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:16.896 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:16.902+0000 7f3195165640 1 -- 172.21.15.84:0/2977951920 >> v1:172.21.15.84:6789/0 conn(0x7f31880c4ec0 legacy=0x7f31880d30a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:16.896 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:16.902+0000 7f3195165640 10 monclient: _finish_hunting 0 2024-03-29T11:28:16.896 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:16.902+0000 7f3195165640 1 monclient: found mon.noname-c 2024-03-29T11:28:16.897 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:16.902+0000 7f3195165640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:28:16.897 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:16.902+0000 7f3195165640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.179:6789/0 2024-03-29T11:28:16.897 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:16.902+0000 7f3195165640 1 -- 172.21.15.84:0/2977951920 --> v1:172.21.15.179:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31880cb410 con 0x7f3188006940 2024-03-29T11:28:16.897 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:16.902+0000 7f3195165640 10 monclient: _finish_auth 0 2024-03-29T11:28:16.897 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:16.902+0000 7f3195165640 10 monclient: _check_auth_tickets 2024-03-29T11:28:16.897 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:16.902+0000 7f3195165640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:28:16.897 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:16.902+0000 7f3195165640 1 -- 172.21.15.84:0/2977951920 <== mon.1 v1:172.21.15.179:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4236868694 0 0) 0x7f3180001690 con 0x7f3188006940 2024-03-29T11:28:16.897 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:16.902+0000 7f3197169640 5 monclient: authenticate success, global_id 4483 2024-03-29T11:28:16.897 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:16.902+0000 7f3197169640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:28:16.897 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:16.902+0000 7f3195165640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:28:16.897 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:16.902+0000 7f3195165640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:28:16.897 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:16.902+0000 7f3195165640 10 monclient: dump: 2024-03-29T11:28:16.897 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T11:28:16.898 INFO:teuthology.misc.health.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:28:16.898 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T11:18:53.864025+0000 2024-03-29T11:28:16.898 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T11:18:53.864025+0000 2024-03-29T11:28:16.898 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T11:28:16.898 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:28:16.898 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T11:28:16.898 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T11:28:16.898 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T11:28:16.898 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:28:16.898 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:16.902+0000 7f3197169640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:28:16.898 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:16.903+0000 7f3195165640 1 -- 172.21.15.84:0/2977951920 <== mon.1 v1:172.21.15.179:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3180001860 con 0x7f3188006940 2024-03-29T11:28:16.898 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:16.903+0000 7f3195165640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:28:16.898 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:16.903+0000 7f3195165640 1 -- 172.21.15.84:0/2977951920 <== mon.1 v1:172.21.15.179:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4236868694 0 0) 0x7f3180001c70 con 0x7f3188006940 2024-03-29T11:28:16.899 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:16.903+0000 7f3197169640 10 monclient: get_monmap_and_config success 2024-03-29T11:28:16.899 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:16.903+0000 7f3197169640 4 set_mon_vals no callback set 2024-03-29T11:28:16.899 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:16.903+0000 7f3195165640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:28:16.899 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:16.903+0000 7f3195165640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:28:16.899 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:16.903+0000 7f3195165640 10 monclient: dump: 2024-03-29T11:28:16.899 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T11:28:16.899 INFO:teuthology.misc.health.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:28:16.899 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T11:18:53.864025+0000 2024-03-29T11:28:16.899 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T11:18:53.864025+0000 2024-03-29T11:28:16.899 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T11:28:16.899 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:28:16.899 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T11:28:16.900 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T11:28:16.900 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T11:28:16.900 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:28:16.900 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:16.903+0000 7f3197169640 10 monclient: shutdown 2024-03-29T11:28:16.900 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:16.903+0000 7f3197169640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:28:16.900 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:16.903+0000 7f3197169640 1 -- 172.21.15.84:0/2977951920 >> v1:172.21.15.179:6789/0 conn(0x7f3188006940 legacy=0x7f31880cad00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:16.900 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:16.904+0000 7f3197169640 1 -- 172.21.15.84:0/2977951920 shutdown_connections 2024-03-29T11:28:16.900 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:16.904+0000 7f3197169640 1 -- 172.21.15.84:0/2977951920 >> 172.21.15.84:0/2977951920 conn(0x7f31880058f0 msgr2=0x7f3188005d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:28:16.900 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:16.904+0000 7f3197169640 1 -- 172.21.15.84:0/2977951920 shutdown_connections 2024-03-29T11:28:16.900 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:16.904+0000 7f3197169640 1 -- 172.21.15.84:0/2977951920 wait complete. 2024-03-29T11:28:16.900 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:16.905+0000 7f3197169640 10 monclient: build_initial_monmap 2024-03-29T11:28:16.900 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:16.905+0000 7f3197169640 10 monclient: monmap: 2024-03-29T11:28:16.900 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-29T11:28:16.901 INFO:teuthology.misc.health.smithi084.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:28:16.901 INFO:teuthology.misc.health.smithi084.stderr:last_changed 0.000000 2024-03-29T11:28:16.901 INFO:teuthology.misc.health.smithi084.stderr:created 0.000000 2024-03-29T11:28:16.901 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-29T11:28:16.901 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:28:16.901 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-29T11:28:16.901 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.noname-b 2024-03-29T11:28:16.901 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-c 2024-03-29T11:28:16.901 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:28:16.906 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:16.905+0000 7f3197169640 1 librados: starting msgr at 2024-03-29T11:28:16.906 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:16.905+0000 7f3197169640 1 librados: starting objecter 2024-03-29T11:28:16.906 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:16.905+0000 7f3197169640 1 Processor -- start 2024-03-29T11:28:16.906 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:16.905+0000 7f3197169640 1 -- start start 2024-03-29T11:28:16.906 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:16.905+0000 7f3197169640 1 librados: setting wanted keys 2024-03-29T11:28:16.906 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:16.905+0000 7f3197169640 1 librados: calling monclient init 2024-03-29T11:28:16.906 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:16.905+0000 7f3197169640 10 monclient: init 2024-03-29T11:28:16.906 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:16.906+0000 7f3197169640 10 monclient: _reopen_session rank -1 2024-03-29T11:28:16.907 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:16.906+0000 7f3197169640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:28:16.907 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:16.906+0000 7f3197169640 10 monclient(hunting): picked mon.noname-c con 0x7f3188006940 addr v1:172.21.15.84:6790/0 2024-03-29T11:28:16.907 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:16.906+0000 7f3197169640 10 monclient(hunting): picked mon.noname-b con 0x7f3188035480 addr v1:172.21.15.179:6789/0 2024-03-29T11:28:16.907 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:16.906+0000 7f3197169640 10 monclient(hunting): picked mon.noname-a con 0x7f31880c4ec0 addr v1:172.21.15.84:6789/0 2024-03-29T11:28:16.907 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:16.906+0000 7f3197169640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f31880c00e0 con 0x7f31880c4ec0 2024-03-29T11:28:16.907 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:16.906+0000 7f3197169640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f318800d240 con 0x7f3188035480 2024-03-29T11:28:16.907 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:16.906+0000 7f3197169640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f318800d520 con 0x7f3188006940 2024-03-29T11:28:16.907 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:16.906+0000 7f3197169640 10 monclient(hunting): _renew_subs 2024-03-29T11:28:16.907 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:16.906+0000 7f3197169640 10 monclient(hunting): authenticate will time out at 1358.201889s 2024-03-29T11:28:16.907 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:16.907+0000 7f3196968640 1 --1- >> v1:172.21.15.84:6789/0 conn(0x7f31880c4ec0 0x7f318816bb60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.84:6789/0 says I am v1:172.21.15.84:56638/0 (socket says 172.21.15.84:56638) 2024-03-29T11:28:16.907 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:16.907+0000 7f3196968640 1 -- 172.21.15.84:0/3941264941 learned_addr learned my addr 172.21.15.84:0/3941264941 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-29T11:28:16.907 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:16.907+0000 7f3186ffd640 10 client.?.objecter ms_handle_connect 0x7f31880c4ec0 2024-03-29T11:28:16.908 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:16.907+0000 7f3186ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:28:16.908 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:16.907+0000 7f3186ffd640 10 client.?.objecter ms_handle_connect 0x7f3188006940 2024-03-29T11:28:16.908 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:16.907+0000 7f3186ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:28:16.908 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:16.907+0000 7f3186ffd640 1 -- 172.21.15.84:0/3941264941 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1485109822 0 0) 0x7f318800d520 con 0x7f3188006940 2024-03-29T11:28:16.908 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:16.907+0000 7f3186ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:28:16.908 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:16.907+0000 7f3186ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:28:16.908 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:16.907+0000 7f3186ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:28:16.908 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:16.907+0000 7f3186ffd640 10 monclient(hunting): my global_id is 4586 2024-03-29T11:28:16.908 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:16.907+0000 7f3186ffd640 1 -- 172.21.15.84:0/3941264941 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3174004eb0 con 0x7f3188006940 2024-03-29T11:28:16.908 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:16.907+0000 7f3186ffd640 1 -- 172.21.15.84:0/3941264941 <== mon.0 v1:172.21.15.84:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2341539490 0 0) 0x7f3188036f30 con 0x7f31880c4ec0 2024-03-29T11:28:16.908 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:16.907+0000 7f3186ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:28:16.908 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:16.907+0000 7f3186ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:28:16.908 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:16.907+0000 7f3186ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:28:16.909 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:16.907+0000 7f3186ffd640 10 monclient(hunting): my global_id is 4533 2024-03-29T11:28:16.909 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:16.907+0000 7f3186ffd640 1 -- 172.21.15.84:0/3941264941 --> v1:172.21.15.84:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f318800d520 con 0x7f31880c4ec0 2024-03-29T11:28:16.909 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:16.908+0000 7f3186ffd640 1 -- 172.21.15.84:0/3941264941 <== mon.0 v1:172.21.15.84:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3705128756 0 0) 0x7f318800d520 con 0x7f31880c4ec0 2024-03-29T11:28:16.909 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:16.908+0000 7f3186ffd640 1 -- 172.21.15.84:0/3941264941 --> v1:172.21.15.84:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3188036f30 con 0x7f31880c4ec0 2024-03-29T11:28:16.909 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:16.908+0000 7f3186ffd640 1 -- 172.21.15.84:0/3941264941 <== mon.2 v1:172.21.15.84:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 94636465 0 0) 0x7f3174004eb0 con 0x7f3188006940 2024-03-29T11:28:16.909 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:16.908+0000 7f3186ffd640 1 -- 172.21.15.84:0/3941264941 --> v1:172.21.15.84:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f31740058a0 con 0x7f3188006940 2024-03-29T11:28:16.909 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:16.908+0000 7f3186ffd640 1 -- 172.21.15.84:0/3941264941 <== mon.2 v1:172.21.15.84:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4236868694 0 0) 0x7f31800019d0 con 0x7f3188006940 2024-03-29T11:28:16.909 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:16.908+0000 7f3186ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:28:16.909 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:16.908+0000 7f3186ffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:28:16.909 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:16.908+0000 7f3186ffd640 10 monclient(hunting): dump: 2024-03-29T11:28:16.909 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T11:28:16.909 INFO:teuthology.misc.health.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:28:16.910 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T11:18:53.864025+0000 2024-03-29T11:28:16.910 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T11:18:53.864025+0000 2024-03-29T11:28:16.910 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T11:28:16.910 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:28:16.910 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T11:28:16.910 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T11:28:16.910 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T11:28:16.910 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:28:16.910 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:16.908+0000 7f3186ffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:28:16.910 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:16.908+0000 7f3186ffd640 1 -- 172.21.15.84:0/3941264941 <== mon.0 v1:172.21.15.84:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4236868694 0 0) 0x7f318c001730 con 0x7f31880c4ec0 2024-03-29T11:28:16.910 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:16.908+0000 7f3186ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:28:16.910 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:16.908+0000 7f3186ffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:28:16.910 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:16.908+0000 7f3186ffd640 10 monclient(hunting): dump: 2024-03-29T11:28:16.911 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T11:28:16.911 INFO:teuthology.misc.health.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:28:16.911 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T11:18:53.864025+0000 2024-03-29T11:28:16.911 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T11:18:53.864025+0000 2024-03-29T11:28:16.911 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T11:28:16.911 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:28:16.911 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T11:28:16.911 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T11:28:16.911 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T11:28:16.911 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:28:16.911 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:16.908+0000 7f3186ffd640 1 -- 172.21.15.84:0/3941264941 <== mon.2 v1:172.21.15.84:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4215486598 0 0) 0x7f31740058a0 con 0x7f3188006940 2024-03-29T11:28:16.911 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:16.908+0000 7f3186ffd640 1 -- 172.21.15.84:0/3941264941 >> v1:172.21.15.179:6789/0 conn(0x7f3188035480 legacy=0x7f318800bec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:16.912 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:16.908+0000 7f3186ffd640 1 -- 172.21.15.84:0/3941264941 >> v1:172.21.15.84:6789/0 conn(0x7f31880c4ec0 legacy=0x7f318816bb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:16.912 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:16.908+0000 7f3186ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:28:16.912 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:16.908+0000 7f3186ffd640 1 monclient: found mon.c 2024-03-29T11:28:16.912 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:16.908+0000 7f3186ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:28:16.912 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:16.908+0000 7f3186ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-29T11:28:16.912 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:16.908+0000 7f3186ffd640 1 -- 172.21.15.84:0/3941264941 --> v1:172.21.15.84:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f318800d800 con 0x7f3188006940 2024-03-29T11:28:16.912 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:16.908+0000 7f3186ffd640 10 monclient: _finish_auth 0 2024-03-29T11:28:16.912 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:16.908+0000 7f3186ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:28:16.912 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:16.908+0000 7f3186ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:28:16.912 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:16.908+0000 7f3197169640 5 monclient: authenticate success, global_id 4586 2024-03-29T11:28:16.912 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:16.908+0000 7f3197169640 10 monclient: _renew_subs 2024-03-29T11:28:16.912 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:16.908+0000 7f3197169640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-29T11:28:16.912 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:16.908+0000 7f3197169640 1 -- 172.21.15.84:0/3941264941 --> v1:172.21.15.84:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f31880bbb60 con 0x7f3188006940 2024-03-29T11:28:16.913 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:16.908+0000 7f3197169640 10 client.4586.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:28:16.913 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:16.908+0000 7f3197169640 10 monclient: _renew_subs 2024-03-29T11:28:16.913 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:16.908+0000 7f3197169640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-29T11:28:16.913 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:16.908+0000 7f3197169640 1 -- 172.21.15.84:0/3941264941 --> v1:172.21.15.84:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f31880c7e30 con 0x7f3188006940 2024-03-29T11:28:16.913 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:16.908+0000 7f3197169640 1 librados: init done 2024-03-29T11:28:16.913 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:16.908+0000 7f3186ffd640 1 -- 172.21.15.84:0/3941264941 <== mon.2 v1:172.21.15.84:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3180001c50 con 0x7f3188006940 2024-03-29T11:28:16.913 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:16.908+0000 7f3186ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:28:16.913 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:16.908+0000 7f3186ffd640 1 -- 172.21.15.84:0/3941264941 <== mon.2 v1:172.21.15.84:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4236868694 0 0) 0x7f3180004d90 con 0x7f3188006940 2024-03-29T11:28:16.914 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:16.908+0000 7f3186ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:28:16.914 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:16.908+0000 7f3186ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:28:16.914 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:16.908+0000 7f3186ffd640 10 monclient: dump: 2024-03-29T11:28:16.914 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T11:28:16.914 INFO:teuthology.misc.health.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:28:16.914 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T11:18:53.864025+0000 2024-03-29T11:28:16.915 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T11:18:53.864025+0000 2024-03-29T11:28:16.915 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T11:28:16.915 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:28:16.915 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T11:28:16.915 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T11:28:16.915 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T11:28:16.915 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:28:16.915 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:16.908+0000 7f3187fff640 4 set_mon_vals no callback set 2024-03-29T11:28:16.915 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:16.910+0000 7f3197169640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:28:16.915 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:16.910+0000 7f3197169640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:28:16.915 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:16.910+0000 7f3197169640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-29T11:28:16.915 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:16.910+0000 7f3197169640 1 -- 172.21.15.84:0/3941264941 --> v1:172.21.15.84:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3188004cd0 con 0x7f3188006940 2024-03-29T11:28:16.916 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:16.911+0000 7f3186ffd640 1 -- 172.21.15.84:0/3941264941 <== mon.2 v1:172.21.15.84:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4033942909 0 0) 0x7f318001ad60 con 0x7f3188006940 2024-03-29T11:28:16.916 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:16.912+0000 7f3195966640 10 monclient: get_auth_request con 0x7f317406e180 auth_method 0 2024-03-29T11:28:16.916 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:16.912+0000 7f3186ffd640 1 -- 172.21.15.84:0/3941264941 <== mon.2 v1:172.21.15.84:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2808139587 0 0) 0x7f3180084a90 con 0x7f3188006940 2024-03-29T11:28:16.916 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:16.912+0000 7f3186ffd640 10 client.4586.objecter ms_dispatch 0x7f3190000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:28:16.916 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:16.912+0000 7f3186ffd640 3 client.4586.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:28:16.916 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:16.912+0000 7f3186ffd640 3 client.4586.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:28:16.922 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:16.912+0000 7f3186ffd640 20 client.4586.objecter dump_active .. 0 homeless 2024-03-29T11:28:16.922 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:16.912+0000 7f3186ffd640 10 client.4586.objecter ms_handle_connect 0x7f317406e180 2024-03-29T11:28:16.923 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:16.919+0000 7f3186ffd640 1 -- 172.21.15.84:0/3941264941 <== mon.2 v1:172.21.15.84:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f31800501a0 con 0x7f3188006940 2024-03-29T11:28:16.923 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:16.919+0000 7f3186ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:28:16.923 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:16.919+0000 7f3186ffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:28:17.270 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:17.280+0000 7f3197169640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:28:17.270 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:17.280+0000 7f3197169640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:28:17.270 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:17.280+0000 7f3197169640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-29T11:28:17.271 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:17.280+0000 7f3197169640 1 -- 172.21.15.84:0/3941264941 --> v1:172.21.15.84:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3188034560 con 0x7f3188006940 2024-03-29T11:28:17.271 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:17.281+0000 7f3186ffd640 1 -- 172.21.15.84:0/3941264941 <== mon.2 v1:172.21.15.84:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4209477549) 0x7f31800522a0 con 0x7f3188006940 2024-03-29T11:28:17.271 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:17.281+0000 7f3186ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:28:17.271 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:17.281+0000 7f3186ffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:28:17.271 INFO:teuthology.misc.health.smithi084.stdout: 2024-03-29T11:28:17.272 INFO:teuthology.misc.health.smithi084.stdout:{"epoch":10,"fsid":"2c3e002f-9eb0-4860-853b-4ef1ef562213","created":"2024-03-29T11:19:32.029169+0000","modified":"2024-03-29T11:19:34.200299+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:34.200299+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"5c39e276-2034-4716-bbc0-57a634eefa92","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"cd580e99-109f-4bf1-88c0-a5987bc8b9ed","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"11e60a5a-051d-453f-a190-66387cb35592","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"2c329fbe-bbc4-4030-bf87-d711cad04679","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f4213d4a-bd36-4433-91ae-d7e2b02481d3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"f0cf44ea-6055-48af-805b-aed858102ff9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"d895eaa0-cc0b-4ee4-9b5a-2dfcd89a60b7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"9bc5a7bc-8f52-4db9-96d6-de5e68d52eaa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:28:17.274 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:17.284+0000 7f3197169640 10 librados: watch_flush enter 2024-03-29T11:28:17.274 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:17.284+0000 7f3197169640 10 librados: watch_flush exit 2024-03-29T11:28:17.274 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:17.284+0000 7f3197169640 20 client.4586.objecter shutdown clearing up homeless session... 2024-03-29T11:28:17.275 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:17.284+0000 7f3197169640 10 client.4586.objecter successfully canceled tick 2024-03-29T11:28:17.275 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:17.284+0000 7f3197169640 1 -- 172.21.15.84:0/3941264941 >> v1:172.21.15.179:6816/2981134980 conn(0x7f317406e180 legacy=0x7f3174070640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:17.275 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:17.284+0000 7f3197169640 10 monclient: shutdown 2024-03-29T11:28:17.275 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:17.284+0000 7f3197169640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:28:17.275 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:17.284+0000 7f3197169640 1 -- 172.21.15.84:0/3941264941 >> v1:172.21.15.84:6790/0 conn(0x7f3188006940 legacy=0x7f318800b7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:17.275 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:17.284+0000 7f3197169640 1 -- 172.21.15.84:0/3941264941 shutdown_connections 2024-03-29T11:28:17.275 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:17.284+0000 7f3197169640 1 -- 172.21.15.84:0/3941264941 >> 172.21.15.84:0/3941264941 conn(0x7f31880058f0 msgr2=0x7f31880c4580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:28:17.275 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:17.284+0000 7f3197169640 1 -- 172.21.15.84:0/3941264941 shutdown_connections 2024-03-29T11:28:17.276 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:17.284+0000 7f3197169640 1 -- 172.21.15.84:0/3941264941 wait complete. 2024-03-29T11:28:17.276 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:17.285+0000 7f3197169640 1 librados: shutdown 2024-03-29T11:28:17.289 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:28:24.100 DEBUG:teuthology.orchestra.run.smithi084:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:28:24.247 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:24.257+0000 7efe64457640 10 monclient: get_monmap_and_config 2024-03-29T11:28:24.248 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:24.257+0000 7efe64457640 10 monclient: build_initial_monmap 2024-03-29T11:28:24.248 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:24.257+0000 7efe64457640 10 monclient: monmap: 2024-03-29T11:28:24.248 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-29T11:28:24.248 INFO:teuthology.misc.health.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:28:24.248 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T11:28:24.258812+0000 2024-03-29T11:28:24.248 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T11:28:24.258812+0000 2024-03-29T11:28:24.248 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-29T11:28:24.248 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:28:24.248 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-29T11:28:24.248 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.noname-c 2024-03-29T11:28:24.248 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-b 2024-03-29T11:28:24.248 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:28:24.249 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:24.259+0000 7efe64457640 1 Processor -- start 2024-03-29T11:28:24.249 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:24.259+0000 7efe64457640 1 -- start start 2024-03-29T11:28:24.250 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:24.259+0000 7efe64457640 10 monclient: init 2024-03-29T11:28:24.250 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:24.260+0000 7efe64457640 10 monclient: _reopen_session rank -1 2024-03-29T11:28:24.250 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:24.260+0000 7efe64457640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:28:24.250 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:24.260+0000 7efe64457640 10 monclient(hunting): picked mon.noname-b con 0x7efe5c1457f0 addr v1:172.21.15.84:6790/0 2024-03-29T11:28:24.253 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:24.263+0000 7efe64457640 10 monclient(hunting): picked mon.noname-c con 0x7efe5c13f650 addr v1:172.21.15.179:6789/0 2024-03-29T11:28:24.254 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:24.264+0000 7efe621f3640 1 --1- >> v1:172.21.15.84:6790/0 conn(0x7efe5c1457f0 0x7efe5c13ef40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.84:6790/0 says I am v1:172.21.15.84:48816/0 (socket says 172.21.15.84:48816) 2024-03-29T11:28:24.254 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:24.264+0000 7efe64457640 10 monclient(hunting): picked mon.noname-a con 0x7efe5c0a9d70 addr v1:172.21.15.84:6789/0 2024-03-29T11:28:24.254 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:24.264+0000 7efe64457640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efe5c0aaef0 con 0x7efe5c0a9d70 2024-03-29T11:28:24.254 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:24.264+0000 7efe64457640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efe5c145bd0 con 0x7efe5c13f650 2024-03-29T11:28:24.254 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:24.264+0000 7efe64457640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efe5c145eb0 con 0x7efe5c1457f0 2024-03-29T11:28:24.254 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:24.264+0000 7efe64457640 10 monclient(hunting): _renew_subs 2024-03-29T11:28:24.254 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:24.264+0000 7efe64457640 10 monclient(hunting): authenticate will time out at 1365.558320s 2024-03-29T11:28:24.254 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:24.264+0000 7efe621f3640 1 -- 172.21.15.84:0/4070817910 learned_addr learned my addr 172.21.15.84:0/4070817910 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-29T11:28:24.255 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:24.265+0000 7efe611f1640 1 -- 172.21.15.84:0/4070817910 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1694956620 0 0) 0x7efe5c145eb0 con 0x7efe5c1457f0 2024-03-29T11:28:24.255 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:24.265+0000 7efe611f1640 10 monclient(hunting): _negotiate 2024-03-29T11:28:24.255 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:24.265+0000 7efe611f1640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:28:24.255 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:24.265+0000 7efe611f1640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:28:24.255 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:24.265+0000 7efe611f1640 10 monclient(hunting): my global_id is 4589 2024-03-29T11:28:24.255 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:24.265+0000 7efe611f1640 1 -- 172.21.15.84:0/4070817910 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efe48003520 con 0x7efe5c1457f0 2024-03-29T11:28:24.255 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:24.265+0000 7efe611f1640 1 -- 172.21.15.84:0/4070817910 <== mon.2 v1:172.21.15.84:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2597214390 0 0) 0x7efe48003520 con 0x7efe5c1457f0 2024-03-29T11:28:24.255 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:24.265+0000 7efe611f1640 1 -- 172.21.15.84:0/4070817910 >> v1:172.21.15.179:6789/0 conn(0x7efe5c13f650 legacy=0x7efe5c0b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:28:24.255 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:24.265+0000 7efe611f1640 1 -- 172.21.15.84:0/4070817910 >> v1:172.21.15.84:6789/0 conn(0x7efe5c0a9d70 legacy=0x7efe5c14d8c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:28:24.257 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:24.267+0000 7efe611f1640 10 monclient: _finish_hunting 0 2024-03-29T11:28:24.257 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:24.267+0000 7efe611f1640 1 monclient: found mon.noname-b 2024-03-29T11:28:24.257 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:24.267+0000 7efe611f1640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:28:24.257 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:24.267+0000 7efe611f1640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.84:6790/0 2024-03-29T11:28:24.257 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:24.267+0000 7efe611f1640 1 -- 172.21.15.84:0/4070817910 --> v1:172.21.15.84:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe5c146340 con 0x7efe5c1457f0 2024-03-29T11:28:24.257 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:24.267+0000 7efe611f1640 10 monclient: _finish_auth 0 2024-03-29T11:28:24.257 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:24.267+0000 7efe611f1640 10 monclient: _check_auth_tickets 2024-03-29T11:28:24.257 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:24.267+0000 7efe611f1640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:28:24.257 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:24.267+0000 7efe611f1640 1 -- 172.21.15.84:0/4070817910 <== mon.2 v1:172.21.15.84:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4236868694 0 0) 0x7efe58001690 con 0x7efe5c1457f0 2024-03-29T11:28:24.257 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:24.267+0000 7efe64457640 5 monclient: authenticate success, global_id 4589 2024-03-29T11:28:24.257 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:24.267+0000 7efe64457640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:28:24.258 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:24.267+0000 7efe611f1640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:28:24.258 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:24.267+0000 7efe611f1640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:28:24.258 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:24.267+0000 7efe611f1640 10 monclient: dump: 2024-03-29T11:28:24.258 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T11:28:24.258 INFO:teuthology.misc.health.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:28:24.258 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T11:18:53.864025+0000 2024-03-29T11:28:24.258 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T11:18:53.864025+0000 2024-03-29T11:28:24.258 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T11:28:24.258 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:28:24.258 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T11:28:24.258 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T11:28:24.258 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T11:28:24.258 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:28:24.258 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:24.267+0000 7efe64457640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:28:24.259 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:24.267+0000 7efe611f1640 1 -- 172.21.15.84:0/4070817910 <== mon.2 v1:172.21.15.84:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7efe580018f0 con 0x7efe5c1457f0 2024-03-29T11:28:24.259 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:24.267+0000 7efe611f1640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:28:24.259 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:24.267+0000 7efe611f1640 1 -- 172.21.15.84:0/4070817910 <== mon.2 v1:172.21.15.84:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4236868694 0 0) 0x7efe58001d40 con 0x7efe5c1457f0 2024-03-29T11:28:24.259 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:24.267+0000 7efe611f1640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:28:24.259 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:24.267+0000 7efe611f1640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:28:24.259 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:24.267+0000 7efe611f1640 10 monclient: dump: 2024-03-29T11:28:24.259 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T11:28:24.259 INFO:teuthology.misc.health.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:28:24.259 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T11:18:53.864025+0000 2024-03-29T11:28:24.259 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T11:18:53.864025+0000 2024-03-29T11:28:24.259 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T11:28:24.259 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:28:24.259 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T11:28:24.259 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T11:28:24.259 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T11:28:24.260 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:28:24.260 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:24.268+0000 7efe64457640 10 monclient: get_monmap_and_config success 2024-03-29T11:28:24.260 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:24.268+0000 7efe64457640 4 set_mon_vals no callback set 2024-03-29T11:28:24.260 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:24.269+0000 7efe64457640 10 monclient: shutdown 2024-03-29T11:28:24.260 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:24.269+0000 7efe64457640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:28:24.260 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:24.269+0000 7efe64457640 1 -- 172.21.15.84:0/4070817910 >> v1:172.21.15.84:6790/0 conn(0x7efe5c1457f0 legacy=0x7efe5c13ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:24.260 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:24.269+0000 7efe64457640 1 -- 172.21.15.84:0/4070817910 shutdown_connections 2024-03-29T11:28:24.260 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:24.269+0000 7efe64457640 1 -- 172.21.15.84:0/4070817910 >> 172.21.15.84:0/4070817910 conn(0x7efe5c066a00 msgr2=0x7efe5c066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:28:24.261 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:24.271+0000 7efe64457640 1 -- 172.21.15.84:0/4070817910 shutdown_connections 2024-03-29T11:28:24.261 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:24.271+0000 7efe64457640 1 -- 172.21.15.84:0/4070817910 wait complete. 2024-03-29T11:28:24.261 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:24.272+0000 7efe64457640 10 monclient: build_initial_monmap 2024-03-29T11:28:24.261 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:24.272+0000 7efe64457640 10 monclient: monmap: 2024-03-29T11:28:24.262 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-29T11:28:24.262 INFO:teuthology.misc.health.smithi084.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:28:24.262 INFO:teuthology.misc.health.smithi084.stderr:last_changed 0.000000 2024-03-29T11:28:24.262 INFO:teuthology.misc.health.smithi084.stderr:created 0.000000 2024-03-29T11:28:24.262 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-29T11:28:24.262 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:28:24.262 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-29T11:28:24.262 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.noname-b 2024-03-29T11:28:24.262 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-c 2024-03-29T11:28:24.262 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:28:24.263 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:24.272+0000 7efe64457640 1 librados: starting msgr at 2024-03-29T11:28:24.263 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:24.272+0000 7efe64457640 1 librados: starting objecter 2024-03-29T11:28:24.263 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:24.272+0000 7efe64457640 1 Processor -- start 2024-03-29T11:28:24.263 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:24.273+0000 7efe64457640 1 -- start start 2024-03-29T11:28:24.263 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:24.273+0000 7efe64457640 1 librados: setting wanted keys 2024-03-29T11:28:24.263 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:24.273+0000 7efe64457640 1 librados: calling monclient init 2024-03-29T11:28:24.263 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:24.273+0000 7efe64457640 10 monclient: init 2024-03-29T11:28:24.266 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:24.273+0000 7efe64457640 10 monclient: _reopen_session rank -1 2024-03-29T11:28:24.266 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:24.273+0000 7efe64457640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:28:24.267 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:24.273+0000 7efe64457640 10 monclient(hunting): picked mon.noname-a con 0x7efe5c0a9d70 addr v1:172.21.15.84:6789/0 2024-03-29T11:28:24.267 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:24.277+0000 7efe64457640 10 monclient(hunting): picked mon.noname-c con 0x7efe5c13f650 addr v1:172.21.15.84:6790/0 2024-03-29T11:28:24.267 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:24.277+0000 7efe64457640 10 monclient(hunting): picked mon.noname-b con 0x7efe5c1374d0 addr v1:172.21.15.179:6789/0 2024-03-29T11:28:24.267 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:24.277+0000 7efe64457640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efe5c1d7f50 con 0x7efe5c0a9d70 2024-03-29T11:28:24.267 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:24.277+0000 7efe64457640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efe5c1d7ae0 con 0x7efe5c1374d0 2024-03-29T11:28:24.267 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:24.277+0000 7efe64457640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efe5c1d7c70 con 0x7efe5c13f650 2024-03-29T11:28:24.267 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:24.277+0000 7efe64457640 10 monclient(hunting): _renew_subs 2024-03-29T11:28:24.267 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:24.277+0000 7efe64457640 10 monclient(hunting): authenticate will time out at 1365.571189s 2024-03-29T11:28:24.268 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:24.277+0000 7efe619f2640 1 --1- >> v1:172.21.15.84:6790/0 conn(0x7efe5c13f650 0x7efe5c136dc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.84:6790/0 says I am v1:172.21.15.84:56408/0 (socket says 172.21.15.84:56408) 2024-03-29T11:28:24.268 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:24.277+0000 7efe619f2640 1 -- 172.21.15.84:0/3136272070 learned_addr learned my addr 172.21.15.84:0/3136272070 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-29T11:28:24.269 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:24.277+0000 7efe52ffd640 10 client.?.objecter ms_handle_connect 0x7efe5c13f650 2024-03-29T11:28:24.269 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:24.277+0000 7efe52ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:28:24.269 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:24.277+0000 7efe52ffd640 10 client.?.objecter ms_handle_connect 0x7efe5c0a9d70 2024-03-29T11:28:24.269 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:24.277+0000 7efe52ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:28:24.269 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:24.277+0000 7efe52ffd640 1 -- 172.21.15.84:0/3136272070 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1269712634 0 0) 0x7efe5c1d7c70 con 0x7efe5c13f650 2024-03-29T11:28:24.269 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:24.277+0000 7efe52ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:28:24.269 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:24.277+0000 7efe52ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:28:24.269 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:24.277+0000 7efe52ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:28:24.269 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:24.277+0000 7efe52ffd640 10 monclient(hunting): my global_id is 4592 2024-03-29T11:28:24.269 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:24.278+0000 7efe52ffd640 1 -- 172.21.15.84:0/3136272070 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efe40004eb0 con 0x7efe5c13f650 2024-03-29T11:28:24.269 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:24.278+0000 7efe52ffd640 1 -- 172.21.15.84:0/3136272070 <== mon.0 v1:172.21.15.84:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 74590811 0 0) 0x7efe5c1d7f50 con 0x7efe5c0a9d70 2024-03-29T11:28:24.269 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:24.278+0000 7efe52ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:28:24.269 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:24.278+0000 7efe52ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:28:24.269 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:24.278+0000 7efe52ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:28:24.270 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:24.278+0000 7efe52ffd640 10 monclient(hunting): my global_id is 4536 2024-03-29T11:28:24.270 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:24.278+0000 7efe52ffd640 1 -- 172.21.15.84:0/3136272070 --> v1:172.21.15.84:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efe5c1d7c70 con 0x7efe5c0a9d70 2024-03-29T11:28:24.270 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:24.278+0000 7efe52ffd640 1 -- 172.21.15.84:0/3136272070 <== mon.2 v1:172.21.15.84:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1813352318 0 0) 0x7efe40004eb0 con 0x7efe5c13f650 2024-03-29T11:28:24.270 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:24.278+0000 7efe52ffd640 1 -- 172.21.15.84:0/3136272070 --> v1:172.21.15.84:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7efe5c1d7f50 con 0x7efe5c13f650 2024-03-29T11:28:24.270 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:24.278+0000 7efe52ffd640 1 -- 172.21.15.84:0/3136272070 <== mon.2 v1:172.21.15.84:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4236868694 0 0) 0x7efe4c003fe0 con 0x7efe5c13f650 2024-03-29T11:28:24.270 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:24.278+0000 7efe52ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:28:24.270 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:24.278+0000 7efe52ffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:28:24.270 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:24.278+0000 7efe52ffd640 10 monclient(hunting): dump: 2024-03-29T11:28:24.270 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T11:28:24.270 INFO:teuthology.misc.health.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:28:24.271 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T11:18:53.864025+0000 2024-03-29T11:28:24.271 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T11:18:53.864025+0000 2024-03-29T11:28:24.271 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T11:28:24.271 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:28:24.271 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T11:28:24.271 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T11:28:24.271 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T11:28:24.271 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:28:24.271 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:24.278+0000 7efe52ffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:28:24.271 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:24.278+0000 7efe52ffd640 1 -- 172.21.15.84:0/3136272070 <== mon.0 v1:172.21.15.84:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1314988125 0 0) 0x7efe5c1d7c70 con 0x7efe5c0a9d70 2024-03-29T11:28:24.271 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:24.278+0000 7efe52ffd640 1 -- 172.21.15.84:0/3136272070 --> v1:172.21.15.84:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7efe40004eb0 con 0x7efe5c0a9d70 2024-03-29T11:28:24.271 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:24.278+0000 7efe52ffd640 1 -- 172.21.15.84:0/3136272070 <== mon.0 v1:172.21.15.84:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4236868694 0 0) 0x7efe58001d40 con 0x7efe5c0a9d70 2024-03-29T11:28:24.271 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:24.278+0000 7efe52ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:28:24.272 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:24.278+0000 7efe52ffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:28:24.272 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:24.278+0000 7efe52ffd640 10 monclient(hunting): dump: 2024-03-29T11:28:24.272 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T11:28:24.272 INFO:teuthology.misc.health.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:28:24.272 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T11:18:53.864025+0000 2024-03-29T11:28:24.272 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T11:18:53.864025+0000 2024-03-29T11:28:24.272 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T11:28:24.272 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:28:24.272 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T11:28:24.272 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T11:28:24.272 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T11:28:24.272 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:28:24.272 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:24.278+0000 7efe52ffd640 1 -- 172.21.15.84:0/3136272070 <== mon.2 v1:172.21.15.84:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1400767534 0 0) 0x7efe5c1d7f50 con 0x7efe5c13f650 2024-03-29T11:28:24.273 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:24.278+0000 7efe52ffd640 1 -- 172.21.15.84:0/3136272070 >> v1:172.21.15.179:6789/0 conn(0x7efe5c1374d0 legacy=0x7efe5c1d3f60 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:28:24.276 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:24.280+0000 7efe52ffd640 1 -- 172.21.15.84:0/3136272070 >> v1:172.21.15.84:6789/0 conn(0x7efe5c0a9d70 legacy=0x7efe5c1d73d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:24.276 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:24.280+0000 7efe52ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:28:24.276 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:24.280+0000 7efe52ffd640 1 monclient: found mon.c 2024-03-29T11:28:24.276 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:24.280+0000 7efe52ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:28:24.276 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:24.280+0000 7efe52ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-29T11:28:24.276 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:24.280+0000 7efe52ffd640 1 -- 172.21.15.84:0/3136272070 --> v1:172.21.15.84:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe5c0b1520 con 0x7efe5c13f650 2024-03-29T11:28:24.276 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:24.280+0000 7efe52ffd640 10 monclient: _finish_auth 0 2024-03-29T11:28:24.277 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:24.280+0000 7efe52ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:28:24.277 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:24.280+0000 7efe52ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:28:24.277 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:24.280+0000 7efe64457640 5 monclient: authenticate success, global_id 4592 2024-03-29T11:28:24.277 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:24.280+0000 7efe64457640 10 monclient: _renew_subs 2024-03-29T11:28:24.277 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:24.280+0000 7efe64457640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-29T11:28:24.277 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:24.280+0000 7efe64457640 1 -- 172.21.15.84:0/3136272070 --> v1:172.21.15.84:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efe5c0b1770 con 0x7efe5c13f650 2024-03-29T11:28:24.277 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:24.281+0000 7efe64457640 10 client.4592.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:28:24.277 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:24.281+0000 7efe64457640 10 monclient: _renew_subs 2024-03-29T11:28:24.277 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:24.281+0000 7efe64457640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-29T11:28:24.277 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:24.281+0000 7efe64457640 1 -- 172.21.15.84:0/3136272070 --> v1:172.21.15.84:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7efe5c1d4850 con 0x7efe5c13f650 2024-03-29T11:28:24.277 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:24.281+0000 7efe64457640 1 librados: init done 2024-03-29T11:28:24.277 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:24.282+0000 7efe64457640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:28:24.277 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:24.282+0000 7efe52ffd640 1 -- 172.21.15.84:0/3136272070 <== mon.2 v1:172.21.15.84:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7efe4c004140 con 0x7efe5c13f650 2024-03-29T11:28:24.277 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:24.282+0000 7efe64457640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:28:24.277 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:24.282+0000 7efe64457640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-29T11:28:24.278 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:24.282+0000 7efe64457640 1 -- 172.21.15.84:0/3136272070 --> v1:172.21.15.84:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efe5c0abb80 con 0x7efe5c13f650 2024-03-29T11:28:24.278 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:24.283+0000 7efe52ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:28:24.278 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:24.283+0000 7efe52ffd640 1 -- 172.21.15.84:0/3136272070 <== mon.2 v1:172.21.15.84:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4236868694 0 0) 0x7efe4c0010b0 con 0x7efe5c13f650 2024-03-29T11:28:24.278 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:24.283+0000 7efe52ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:28:24.278 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:24.283+0000 7efe52ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:28:24.278 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:24.283+0000 7efe52ffd640 10 monclient: dump: 2024-03-29T11:28:24.278 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T11:28:24.278 INFO:teuthology.misc.health.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:28:24.278 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T11:18:53.864025+0000 2024-03-29T11:28:24.278 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T11:18:53.864025+0000 2024-03-29T11:28:24.278 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T11:28:24.278 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:28:24.278 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T11:28:24.278 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T11:28:24.279 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T11:28:24.279 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:28:24.279 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:24.283+0000 7efe609f0640 4 set_mon_vals no callback set 2024-03-29T11:28:24.280 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:24.287+0000 7efe52ffd640 1 -- 172.21.15.84:0/3136272070 <== mon.2 v1:172.21.15.84:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4033942909 0 0) 0x7efe4c001210 con 0x7efe5c13f650 2024-03-29T11:28:24.280 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:24.288+0000 7efe52ffd640 1 -- 172.21.15.84:0/3136272070 <== mon.2 v1:172.21.15.84:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2808139587 0 0) 0x7efe4c084960 con 0x7efe5c13f650 2024-03-29T11:28:24.280 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:24.288+0000 7efe52ffd640 10 client.4592.objecter ms_dispatch 0x7efe5c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:28:24.280 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:24.288+0000 7efe52ffd640 3 client.4592.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:28:24.280 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:24.288+0000 7efe52ffd640 3 client.4592.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:28:24.280 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:24.288+0000 7efe52ffd640 20 client.4592.objecter dump_active .. 0 homeless 2024-03-29T11:28:24.280 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:24.289+0000 7efe621f3640 10 monclient: get_auth_request con 0x7efe4006de00 auth_method 0 2024-03-29T11:28:24.280 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:24.289+0000 7efe52ffd640 10 client.4592.objecter ms_handle_connect 0x7efe4006de00 2024-03-29T11:28:24.281 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:24.291+0000 7efe52ffd640 1 -- 172.21.15.84:0/3136272070 <== mon.2 v1:172.21.15.84:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7efe4c050070 con 0x7efe5c13f650 2024-03-29T11:28:24.281 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:24.291+0000 7efe52ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:28:24.281 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:24.291+0000 7efe52ffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:28:24.612 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:24.622+0000 7efe64457640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:28:24.612 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:24.622+0000 7efe64457640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:28:24.612 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:24.622+0000 7efe64457640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-29T11:28:24.613 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:24.622+0000 7efe64457640 1 -- 172.21.15.84:0/3136272070 --> v1:172.21.15.84:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7efe5c075f10 con 0x7efe5c13f650 2024-03-29T11:28:24.614 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:24.625+0000 7efe52ffd640 1 -- 172.21.15.84:0/3136272070 <== mon.2 v1:172.21.15.84:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4209477549) 0x7efe4c052170 con 0x7efe5c13f650 2024-03-29T11:28:24.614 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:24.625+0000 7efe52ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:28:24.614 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:24.625+0000 7efe52ffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:28:24.616 INFO:teuthology.misc.health.smithi084.stdout: 2024-03-29T11:28:24.617 INFO:teuthology.misc.health.smithi084.stdout:{"epoch":10,"fsid":"2c3e002f-9eb0-4860-853b-4ef1ef562213","created":"2024-03-29T11:19:32.029169+0000","modified":"2024-03-29T11:19:34.200299+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:34.200299+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"5c39e276-2034-4716-bbc0-57a634eefa92","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"cd580e99-109f-4bf1-88c0-a5987bc8b9ed","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"11e60a5a-051d-453f-a190-66387cb35592","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"2c329fbe-bbc4-4030-bf87-d711cad04679","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f4213d4a-bd36-4433-91ae-d7e2b02481d3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"f0cf44ea-6055-48af-805b-aed858102ff9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"d895eaa0-cc0b-4ee4-9b5a-2dfcd89a60b7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"9bc5a7bc-8f52-4db9-96d6-de5e68d52eaa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:28:24.620 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:24.630+0000 7efe64457640 10 librados: watch_flush enter 2024-03-29T11:28:24.620 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:24.630+0000 7efe64457640 10 librados: watch_flush exit 2024-03-29T11:28:24.620 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:24.630+0000 7efe64457640 20 client.4592.objecter shutdown clearing up homeless session... 2024-03-29T11:28:24.621 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:24.630+0000 7efe64457640 10 client.4592.objecter successfully canceled tick 2024-03-29T11:28:24.621 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:24.630+0000 7efe64457640 1 -- 172.21.15.84:0/3136272070 >> v1:172.21.15.179:6816/2981134980 conn(0x7efe4006de00 legacy=0x7efe400702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:24.621 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:24.630+0000 7efe64457640 10 monclient: shutdown 2024-03-29T11:28:24.621 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:24.630+0000 7efe64457640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:28:24.621 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:24.630+0000 7efe64457640 1 -- 172.21.15.84:0/3136272070 >> v1:172.21.15.84:6790/0 conn(0x7efe5c13f650 legacy=0x7efe5c136dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:24.621 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:24.631+0000 7efe64457640 1 -- 172.21.15.84:0/3136272070 shutdown_connections 2024-03-29T11:28:24.621 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:24.631+0000 7efe64457640 1 -- 172.21.15.84:0/3136272070 >> 172.21.15.84:0/3136272070 conn(0x7efe5c066a00 msgr2=0x7efe5c0b3a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:28:24.622 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:24.631+0000 7efe64457640 1 -- 172.21.15.84:0/3136272070 shutdown_connections 2024-03-29T11:28:24.622 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:24.631+0000 7efe64457640 1 -- 172.21.15.84:0/3136272070 wait complete. 2024-03-29T11:28:24.622 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:24.632+0000 7efe64457640 1 librados: shutdown 2024-03-29T11:28:24.640 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:28:31.448 DEBUG:teuthology.orchestra.run.smithi084:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:28:31.594 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:31.604+0000 7f316f59e640 10 monclient: get_monmap_and_config 2024-03-29T11:28:31.595 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:31.604+0000 7f316f59e640 10 monclient: build_initial_monmap 2024-03-29T11:28:31.595 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:31.604+0000 7f316f59e640 10 monclient: monmap: 2024-03-29T11:28:31.595 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-29T11:28:31.595 INFO:teuthology.misc.health.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:28:31.595 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T11:28:31.605683+0000 2024-03-29T11:28:31.595 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T11:28:31.605683+0000 2024-03-29T11:28:31.595 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-29T11:28:31.595 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:28:31.595 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-29T11:28:31.595 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.noname-c 2024-03-29T11:28:31.595 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-b 2024-03-29T11:28:31.595 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:28:31.597 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:31.605+0000 7f316f59e640 1 Processor -- start 2024-03-29T11:28:31.599 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:31.605+0000 7f316f59e640 1 -- start start 2024-03-29T11:28:31.599 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:31.605+0000 7f316f59e640 10 monclient: init 2024-03-29T11:28:31.599 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:31.606+0000 7f316f59e640 10 monclient: _reopen_session rank -1 2024-03-29T11:28:31.600 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:31.606+0000 7f316f59e640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:28:31.600 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:31.606+0000 7f316f59e640 10 monclient(hunting): picked mon.noname-b con 0x7f3168006e30 addr v1:172.21.15.84:6790/0 2024-03-29T11:28:31.600 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:31.606+0000 7f316f59e640 10 monclient(hunting): picked mon.noname-c con 0x7f3168035e00 addr v1:172.21.15.179:6789/0 2024-03-29T11:28:31.600 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:31.606+0000 7f316f59e640 10 monclient(hunting): picked mon.noname-a con 0x7f31680c9670 addr v1:172.21.15.84:6789/0 2024-03-29T11:28:31.600 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:31.606+0000 7f316f59e640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3168035970 con 0x7f31680c9670 2024-03-29T11:28:31.600 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:31.606+0000 7f316f59e640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3168036280 con 0x7f3168035e00 2024-03-29T11:28:31.600 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:31.606+0000 7f316f59e640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3168035640 con 0x7f3168006e30 2024-03-29T11:28:31.600 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:31.606+0000 7f316f59e640 10 monclient(hunting): _renew_subs 2024-03-29T11:28:31.600 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:31.606+0000 7f316f59e640 10 monclient(hunting): authenticate will time out at 1372.901048s 2024-03-29T11:28:31.600 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:31.607+0000 7f316e59c640 1 --1- >> v1:172.21.15.84:6790/0 conn(0x7f3168006e30 0x7f31680cbcc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.84:6790/0 says I am v1:172.21.15.84:46512/0 (socket says 172.21.15.84:46512) 2024-03-29T11:28:31.600 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:31.607+0000 7f316e59c640 1 -- 172.21.15.84:0/2331822267 learned_addr learned my addr 172.21.15.84:0/2331822267 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-29T11:28:31.601 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:31.610+0000 7f316d59a640 1 -- 172.21.15.84:0/2331822267 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 263347293 0 0) 0x7f3168035640 con 0x7f3168006e30 2024-03-29T11:28:31.601 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:31.610+0000 7f316d59a640 10 monclient(hunting): _negotiate 2024-03-29T11:28:31.601 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:31.610+0000 7f316d59a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:28:31.601 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:31.610+0000 7f316d59a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:28:31.601 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:31.610+0000 7f316d59a640 10 monclient(hunting): my global_id is 4595 2024-03-29T11:28:31.602 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:31.610+0000 7f316d59a640 1 -- 172.21.15.84:0/2331822267 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3158003520 con 0x7f3168006e30 2024-03-29T11:28:31.602 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:31.610+0000 7f316d59a640 1 -- 172.21.15.84:0/2331822267 <== mon.1 v1:172.21.15.179:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 186605523 0 0) 0x7f3168036280 con 0x7f3168035e00 2024-03-29T11:28:31.602 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:31.610+0000 7f316d59a640 10 monclient(hunting): _negotiate 2024-03-29T11:28:31.602 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:31.610+0000 7f316d59a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:28:31.602 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:31.610+0000 7f316d59a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:28:31.602 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:31.610+0000 7f316d59a640 10 monclient(hunting): my global_id is 4486 2024-03-29T11:28:31.602 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:31.610+0000 7f316d59a640 1 -- 172.21.15.84:0/2331822267 --> v1:172.21.15.179:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3168035640 con 0x7f3168035e00 2024-03-29T11:28:31.602 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:31.611+0000 7f316d59a640 1 -- 172.21.15.84:0/2331822267 <== mon.1 v1:172.21.15.179:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 804279300 0 0) 0x7f3168035640 con 0x7f3168035e00 2024-03-29T11:28:31.602 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:31.611+0000 7f316d59a640 1 -- 172.21.15.84:0/2331822267 >> v1:172.21.15.84:6790/0 conn(0x7f3168006e30 legacy=0x7f31680cbcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:31.602 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:31.611+0000 7f316d59a640 1 -- 172.21.15.84:0/2331822267 >> v1:172.21.15.84:6789/0 conn(0x7f31680c9670 legacy=0x7f31680d7560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:28:31.602 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:31.611+0000 7f316d59a640 10 monclient: _finish_hunting 0 2024-03-29T11:28:31.602 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:31.611+0000 7f316d59a640 1 monclient: found mon.noname-c 2024-03-29T11:28:31.602 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:31.611+0000 7f316d59a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:28:31.603 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:31.611+0000 7f316d59a640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.179:6789/0 2024-03-29T11:28:31.603 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:31.611+0000 7f316d59a640 1 -- 172.21.15.84:0/2331822267 --> v1:172.21.15.179:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3168034e20 con 0x7f3168035e00 2024-03-29T11:28:31.603 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:31.611+0000 7f316d59a640 10 monclient: _finish_auth 0 2024-03-29T11:28:31.603 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:31.611+0000 7f316d59a640 10 monclient: _check_auth_tickets 2024-03-29T11:28:31.603 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:31.611+0000 7f316d59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:28:31.603 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:31.611+0000 7f316d59a640 1 -- 172.21.15.84:0/2331822267 <== mon.1 v1:172.21.15.179:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4236868694 0 0) 0x7f3160001690 con 0x7f3168035e00 2024-03-29T11:28:31.603 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:31.611+0000 7f316d59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:28:31.603 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:31.611+0000 7f316d59a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:28:31.603 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:31.611+0000 7f316d59a640 10 monclient: dump: 2024-03-29T11:28:31.603 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T11:28:31.603 INFO:teuthology.misc.health.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:28:31.603 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T11:18:53.864025+0000 2024-03-29T11:28:31.603 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T11:18:53.864025+0000 2024-03-29T11:28:31.603 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T11:28:31.603 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:28:31.604 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T11:28:31.604 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T11:28:31.604 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T11:28:31.604 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:28:31.604 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:31.611+0000 7f316f59e640 5 monclient: authenticate success, global_id 4486 2024-03-29T11:28:31.604 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:31.611+0000 7f316f59e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:28:31.604 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:31.611+0000 7f316d59a640 1 -- 172.21.15.84:0/2331822267 <== mon.1 v1:172.21.15.179:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3160001860 con 0x7f3168035e00 2024-03-29T11:28:31.604 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:31.611+0000 7f316d59a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:28:31.604 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:31.611+0000 7f316d59a640 1 -- 172.21.15.84:0/2331822267 <== mon.1 v1:172.21.15.179:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4236868694 0 0) 0x7f3160001cb0 con 0x7f3168035e00 2024-03-29T11:28:31.604 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:31.611+0000 7f316d59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:28:31.604 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:31.611+0000 7f316d59a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:28:31.604 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:31.611+0000 7f316d59a640 10 monclient: dump: 2024-03-29T11:28:31.604 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T11:28:31.604 INFO:teuthology.misc.health.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:28:31.605 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T11:18:53.864025+0000 2024-03-29T11:28:31.605 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T11:18:53.864025+0000 2024-03-29T11:28:31.605 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T11:28:31.605 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:28:31.605 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T11:28:31.605 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T11:28:31.605 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T11:28:31.605 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:28:31.605 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:31.611+0000 7f316f59e640 10 monclient: get_monmap_and_config success 2024-03-29T11:28:31.605 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:31.611+0000 7f316f59e640 4 set_mon_vals no callback set 2024-03-29T11:28:31.609 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:31.612+0000 7f316f59e640 10 monclient: shutdown 2024-03-29T11:28:31.609 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:31.612+0000 7f316f59e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:28:31.610 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:31.612+0000 7f316f59e640 1 -- 172.21.15.84:0/2331822267 >> v1:172.21.15.179:6789/0 conn(0x7f3168035e00 legacy=0x7f31680c8f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:31.610 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:31.613+0000 7f316f59e640 1 -- 172.21.15.84:0/2331822267 shutdown_connections 2024-03-29T11:28:31.610 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:31.613+0000 7f316f59e640 1 -- 172.21.15.84:0/2331822267 >> 172.21.15.84:0/2331822267 conn(0x7f3168005c00 msgr2=0x7f3168006010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:28:31.610 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:31.613+0000 7f316f59e640 1 -- 172.21.15.84:0/2331822267 shutdown_connections 2024-03-29T11:28:31.610 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:31.613+0000 7f316f59e640 1 -- 172.21.15.84:0/2331822267 wait complete. 2024-03-29T11:28:31.610 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:31.614+0000 7f316f59e640 10 monclient: build_initial_monmap 2024-03-29T11:28:31.610 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:31.614+0000 7f316f59e640 10 monclient: monmap: 2024-03-29T11:28:31.610 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-29T11:28:31.610 INFO:teuthology.misc.health.smithi084.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:28:31.610 INFO:teuthology.misc.health.smithi084.stderr:last_changed 0.000000 2024-03-29T11:28:31.610 INFO:teuthology.misc.health.smithi084.stderr:created 0.000000 2024-03-29T11:28:31.610 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-29T11:28:31.610 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:28:31.610 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-29T11:28:31.611 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.noname-b 2024-03-29T11:28:31.611 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-c 2024-03-29T11:28:31.611 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:28:31.611 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:31.614+0000 7f316f59e640 1 librados: starting msgr at 2024-03-29T11:28:31.611 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:31.614+0000 7f316f59e640 1 librados: starting objecter 2024-03-29T11:28:31.611 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:31.615+0000 7f316f59e640 1 Processor -- start 2024-03-29T11:28:31.611 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:31.615+0000 7f316f59e640 1 -- start start 2024-03-29T11:28:31.611 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:31.615+0000 7f316f59e640 1 librados: setting wanted keys 2024-03-29T11:28:31.611 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:31.615+0000 7f316f59e640 1 librados: calling monclient init 2024-03-29T11:28:31.611 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:31.615+0000 7f316f59e640 10 monclient: init 2024-03-29T11:28:31.611 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:31.615+0000 7f316f59e640 10 monclient: _reopen_session rank -1 2024-03-29T11:28:31.611 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:31.615+0000 7f316f59e640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:28:31.611 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:31.616+0000 7f316f59e640 10 monclient(hunting): picked mon.noname-a con 0x7f3168006e30 addr v1:172.21.15.84:6789/0 2024-03-29T11:28:31.611 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:31.616+0000 7f316f59e640 10 monclient(hunting): picked mon.noname-b con 0x7f3168035e00 addr v1:172.21.15.179:6789/0 2024-03-29T11:28:31.611 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:31.616+0000 7f316f59e640 10 monclient(hunting): picked mon.noname-c con 0x7f31680c9670 addr v1:172.21.15.84:6790/0 2024-03-29T11:28:31.612 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:31.616+0000 7f316f59e640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f31680cf490 con 0x7f3168006e30 2024-03-29T11:28:31.612 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:31.616+0000 7f316f59e640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f31680cf920 con 0x7f3168035e00 2024-03-29T11:28:31.612 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:31.616+0000 7f316f59e640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f31680ce730 con 0x7f31680c9670 2024-03-29T11:28:31.612 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:31.616+0000 7f316f59e640 10 monclient(hunting): _renew_subs 2024-03-29T11:28:31.612 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:31.616+0000 7f316f59e640 10 monclient(hunting): authenticate will time out at 1372.910961s 2024-03-29T11:28:31.612 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:31.617+0000 7f316ed9d640 1 --1- >> v1:172.21.15.84:6790/0 conn(0x7f31680c9670 0x7f3168169990 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.84:6790/0 says I am v1:172.21.15.84:52002/0 (socket says 172.21.15.84:52002) 2024-03-29T11:28:31.612 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:31.617+0000 7f316ed9d640 1 -- 172.21.15.84:0/3428245009 learned_addr learned my addr 172.21.15.84:0/3428245009 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-29T11:28:31.612 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:31.617+0000 7f315effd640 10 client.?.objecter ms_handle_connect 0x7f31680c9670 2024-03-29T11:28:31.612 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:31.617+0000 7f315effd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:28:31.612 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:31.618+0000 7f315effd640 1 -- 172.21.15.84:0/3428245009 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3379439786 0 0) 0x7f31680ce730 con 0x7f31680c9670 2024-03-29T11:28:31.612 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:31.618+0000 7f315effd640 10 monclient(hunting): _negotiate 2024-03-29T11:28:31.612 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:31.618+0000 7f315effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:28:31.612 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:31.618+0000 7f315effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:28:31.612 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:31.618+0000 7f315effd640 10 monclient(hunting): my global_id is 4598 2024-03-29T11:28:31.612 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:31.618+0000 7f315effd640 1 -- 172.21.15.84:0/3428245009 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3150004eb0 con 0x7f31680c9670 2024-03-29T11:28:31.613 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:31.618+0000 7f315effd640 1 -- 172.21.15.84:0/3428245009 <== mon.2 v1:172.21.15.84:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3996431502 0 0) 0x7f3150004eb0 con 0x7f31680c9670 2024-03-29T11:28:31.613 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:31.618+0000 7f315effd640 1 -- 172.21.15.84:0/3428245009 --> v1:172.21.15.84:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f31680ce730 con 0x7f31680c9670 2024-03-29T11:28:31.613 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:31.618+0000 7f315effd640 1 -- 172.21.15.84:0/3428245009 <== mon.2 v1:172.21.15.84:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4236868694 0 0) 0x7f3164001750 con 0x7f31680c9670 2024-03-29T11:28:31.613 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:31.618+0000 7f315effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:28:31.613 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:31.618+0000 7f315effd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:28:31.613 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:31.618+0000 7f315effd640 10 monclient(hunting): dump: 2024-03-29T11:28:31.613 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T11:28:31.613 INFO:teuthology.misc.health.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:28:31.613 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T11:18:53.864025+0000 2024-03-29T11:28:31.613 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T11:18:53.864025+0000 2024-03-29T11:28:31.613 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T11:28:31.613 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:28:31.613 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T11:28:31.613 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T11:28:31.613 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T11:28:31.614 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:28:31.614 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:31.618+0000 7f315effd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:28:31.614 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:31.618+0000 7f315effd640 1 -- 172.21.15.84:0/3428245009 <== mon.2 v1:172.21.15.84:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1256529617 0 0) 0x7f31680ce730 con 0x7f31680c9670 2024-03-29T11:28:31.614 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:31.618+0000 7f315effd640 1 -- 172.21.15.84:0/3428245009 >> v1:172.21.15.179:6789/0 conn(0x7f3168035e00 legacy=0x7f3168160420 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:28:31.617 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:31.620+0000 7f315effd640 1 -- 172.21.15.84:0/3428245009 >> v1:172.21.15.84:6789/0 conn(0x7f3168006e30 legacy=0x7f31680cead0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:31.617 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:31.620+0000 7f315effd640 10 monclient: _finish_hunting 0 2024-03-29T11:28:31.617 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:31.620+0000 7f315effd640 1 monclient: found mon.c 2024-03-29T11:28:31.617 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:31.620+0000 7f315effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:28:31.618 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:31.620+0000 7f315effd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-29T11:28:31.618 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:31.620+0000 7f315effd640 1 -- 172.21.15.84:0/3428245009 --> v1:172.21.15.84:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31680ce8c0 con 0x7f31680c9670 2024-03-29T11:28:31.618 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:31.620+0000 7f315effd640 10 monclient: _finish_auth 0 2024-03-29T11:28:31.618 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:31.620+0000 7f315effd640 10 monclient: _check_auth_tickets 2024-03-29T11:28:31.618 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:31.620+0000 7f315effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:28:31.618 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:31.620+0000 7f316f59e640 5 monclient: authenticate success, global_id 4598 2024-03-29T11:28:31.618 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:31.620+0000 7f316f59e640 10 monclient: _renew_subs 2024-03-29T11:28:31.618 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:31.620+0000 7f316f59e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-29T11:28:31.618 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:31.620+0000 7f316f59e640 1 -- 172.21.15.84:0/3428245009 --> v1:172.21.15.84:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3168032fc0 con 0x7f31680c9670 2024-03-29T11:28:31.619 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:31.620+0000 7f316f59e640 10 client.4598.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:28:31.619 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:31.620+0000 7f316f59e640 10 monclient: _renew_subs 2024-03-29T11:28:31.619 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:31.620+0000 7f316f59e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-29T11:28:31.619 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:31.620+0000 7f316f59e640 1 -- 172.21.15.84:0/3428245009 --> v1:172.21.15.84:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f3168160b30 con 0x7f31680c9670 2024-03-29T11:28:31.619 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:31.620+0000 7f316f59e640 1 librados: init done 2024-03-29T11:28:31.619 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:31.621+0000 7f315effd640 10 client.4598.objecter ms_handle_connect 0x7f3168006e30 2024-03-29T11:28:31.619 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:31.621+0000 7f315effd640 10 client.4598.objecter resend_mon_ops 2024-03-29T11:28:31.619 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:31.621+0000 7f315effd640 1 -- 172.21.15.84:0/3428245009 <== mon.2 v1:172.21.15.84:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f31640019b0 con 0x7f31680c9670 2024-03-29T11:28:31.619 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:31.621+0000 7f315effd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:28:31.619 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:31.621+0000 7f315effd640 1 -- 172.21.15.84:0/3428245009 <== mon.2 v1:172.21.15.84:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4236868694 0 0) 0x7f3164004a40 con 0x7f31680c9670 2024-03-29T11:28:31.619 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:31.621+0000 7f315effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:28:31.619 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:31.621+0000 7f315effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:28:31.619 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:31.621+0000 7f315effd640 10 monclient: dump: 2024-03-29T11:28:31.619 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T11:28:31.619 INFO:teuthology.misc.health.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:28:31.620 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T11:18:53.864025+0000 2024-03-29T11:28:31.620 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T11:18:53.864025+0000 2024-03-29T11:28:31.620 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T11:28:31.620 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:28:31.620 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T11:28:31.620 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T11:28:31.620 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T11:28:31.620 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:28:31.620 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:31.622+0000 7f315cff9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:28:31.620 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:31.622+0000 7f315cff9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:28:31.620 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:31.622+0000 7f315cff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-29T11:28:31.620 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:31.622+0000 7f315cff9640 1 -- 172.21.15.84:0/3428245009 --> v1:172.21.15.84:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3168007b90 con 0x7f31680c9670 2024-03-29T11:28:31.620 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:31.622+0000 7f316cd99640 4 set_mon_vals no callback set 2024-03-29T11:28:31.620 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:31.625+0000 7f315effd640 1 -- 172.21.15.84:0/3428245009 <== mon.2 v1:172.21.15.84:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4033942909 0 0) 0x7f3164004ba0 con 0x7f31680c9670 2024-03-29T11:28:31.621 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:31.628+0000 7f315effd640 1 -- 172.21.15.84:0/3428245009 <== mon.2 v1:172.21.15.84:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2808139587 0 0) 0x7f3164084a10 con 0x7f31680c9670 2024-03-29T11:28:31.621 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:31.628+0000 7f315effd640 10 client.4598.objecter ms_dispatch 0x7f3170000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:28:31.621 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:31.628+0000 7f315effd640 3 client.4598.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:28:31.621 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:31.628+0000 7f315effd640 3 client.4598.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:28:31.621 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:31.628+0000 7f316e59c640 10 monclient: get_auth_request con 0x7f315006da40 auth_method 0 2024-03-29T11:28:31.621 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:31.628+0000 7f315effd640 20 client.4598.objecter dump_active .. 0 homeless 2024-03-29T11:28:31.621 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:31.629+0000 7f315effd640 10 client.4598.objecter ms_handle_connect 0x7f315006da40 2024-03-29T11:28:31.622 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:31.632+0000 7f315effd640 1 -- 172.21.15.84:0/3428245009 <== mon.2 v1:172.21.15.84:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f3164050120 con 0x7f31680c9670 2024-03-29T11:28:31.622 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:31.632+0000 7f315effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:28:31.622 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:31.632+0000 7f315effd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:28:31.949 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:31.958+0000 7f315cff9640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:28:31.950 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:31.958+0000 7f315cff9640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:28:31.950 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:31.958+0000 7f315cff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-29T11:28:31.950 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:31.958+0000 7f315cff9640 1 -- 172.21.15.84:0/3428245009 --> v1:172.21.15.84:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f31680cf260 con 0x7f31680c9670 2024-03-29T11:28:31.950 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:31.961+0000 7f315effd640 1 -- 172.21.15.84:0/3428245009 <== mon.2 v1:172.21.15.84:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4209477549) 0x7f31680cf260 con 0x7f31680c9670 2024-03-29T11:28:31.951 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:31.961+0000 7f315effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:28:31.951 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:31.961+0000 7f315effd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:28:31.954 INFO:teuthology.misc.health.smithi084.stdout: 2024-03-29T11:28:31.954 INFO:teuthology.misc.health.smithi084.stdout:{"epoch":10,"fsid":"2c3e002f-9eb0-4860-853b-4ef1ef562213","created":"2024-03-29T11:19:32.029169+0000","modified":"2024-03-29T11:19:34.200299+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:34.200299+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"5c39e276-2034-4716-bbc0-57a634eefa92","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"cd580e99-109f-4bf1-88c0-a5987bc8b9ed","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"11e60a5a-051d-453f-a190-66387cb35592","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"2c329fbe-bbc4-4030-bf87-d711cad04679","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f4213d4a-bd36-4433-91ae-d7e2b02481d3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"f0cf44ea-6055-48af-805b-aed858102ff9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"d895eaa0-cc0b-4ee4-9b5a-2dfcd89a60b7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"9bc5a7bc-8f52-4db9-96d6-de5e68d52eaa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:28:31.958 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:31.968+0000 7f315cff9640 10 librados: watch_flush enter 2024-03-29T11:28:31.958 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:31.968+0000 7f315cff9640 10 librados: watch_flush exit 2024-03-29T11:28:31.958 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:31.968+0000 7f315cff9640 20 client.4598.objecter shutdown clearing up homeless session... 2024-03-29T11:28:31.958 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:31.968+0000 7f315cff9640 10 client.4598.objecter successfully canceled tick 2024-03-29T11:28:31.958 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:31.968+0000 7f315cff9640 1 -- 172.21.15.84:0/3428245009 >> v1:172.21.15.179:6816/2981134980 conn(0x7f315006da40 legacy=0x7f315006ff00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:31.958 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:31.968+0000 7f315cff9640 10 monclient: shutdown 2024-03-29T11:28:31.958 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:31.968+0000 7f315cff9640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:28:31.958 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:31.968+0000 7f315cff9640 1 -- 172.21.15.84:0/3428245009 >> v1:172.21.15.84:6790/0 conn(0x7f31680c9670 legacy=0x7f3168169990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:31.961 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:31.971+0000 7f315cff9640 1 -- 172.21.15.84:0/3428245009 shutdown_connections 2024-03-29T11:28:31.961 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:31.971+0000 7f315cff9640 1 -- 172.21.15.84:0/3428245009 >> 172.21.15.84:0/3428245009 conn(0x7f3168005c00 msgr2=0x7f31680c8d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:28:31.963 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:31.973+0000 7f315cff9640 1 -- 172.21.15.84:0/3428245009 shutdown_connections 2024-03-29T11:28:31.965 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:31.976+0000 7f315cff9640 1 -- 172.21.15.84:0/3428245009 wait complete. 2024-03-29T11:28:31.965 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:31.976+0000 7f315cff9640 1 librados: shutdown 2024-03-29T11:28:31.986 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:28:38.795 DEBUG:teuthology.orchestra.run.smithi084:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:28:38.942 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:38.951+0000 7f544748f640 10 monclient: get_monmap_and_config 2024-03-29T11:28:38.942 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:38.951+0000 7f544748f640 10 monclient: build_initial_monmap 2024-03-29T11:28:38.942 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:38.951+0000 7f544748f640 10 monclient: monmap: 2024-03-29T11:28:38.942 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-29T11:28:38.942 INFO:teuthology.misc.health.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:28:38.942 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T11:28:38.952705+0000 2024-03-29T11:28:38.942 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T11:28:38.952705+0000 2024-03-29T11:28:38.942 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-29T11:28:38.942 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:28:38.942 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-29T11:28:38.942 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.noname-c 2024-03-29T11:28:38.942 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-b 2024-03-29T11:28:38.943 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:28:38.943 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:38.952+0000 7f544748f640 1 Processor -- start 2024-03-29T11:28:38.943 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:38.952+0000 7f544748f640 1 -- start start 2024-03-29T11:28:38.944 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:38.952+0000 7f544748f640 10 monclient: init 2024-03-29T11:28:38.944 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:38.953+0000 7f544748f640 10 monclient: _reopen_session rank -1 2024-03-29T11:28:38.944 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:38.953+0000 7f544748f640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:28:38.944 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:38.954+0000 7f544748f640 10 monclient(hunting): picked mon.noname-b con 0x7f54380389e0 addr v1:172.21.15.84:6790/0 2024-03-29T11:28:38.944 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:38.954+0000 7f544648d640 1 --1- >> v1:172.21.15.84:6790/0 conn(0x7f54380389e0 0x7f5438033520 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.84:6790/0 says I am v1:172.21.15.84:44472/0 (socket says 172.21.15.84:44472) 2024-03-29T11:28:38.944 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:38.954+0000 7f544748f640 10 monclient(hunting): picked mon.noname-c con 0x7f5438038dc0 addr v1:172.21.15.179:6789/0 2024-03-29T11:28:38.944 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:38.954+0000 7f544748f640 10 monclient(hunting): picked mon.noname-a con 0x7f5438030660 addr v1:172.21.15.84:6789/0 2024-03-29T11:28:38.944 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:38.954+0000 7f544748f640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54380c54c0 con 0x7f5438030660 2024-03-29T11:28:38.945 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:38.954+0000 7f544748f640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5438036430 con 0x7f5438038dc0 2024-03-29T11:28:38.945 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:38.955+0000 7f544748f640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5438039840 con 0x7f54380389e0 2024-03-29T11:28:38.945 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:38.955+0000 7f544748f640 10 monclient(hunting): _renew_subs 2024-03-29T11:28:38.945 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:38.955+0000 7f544748f640 10 monclient(hunting): authenticate will time out at 1380.249122s 2024-03-29T11:28:38.947 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:38.957+0000 7f544648d640 1 -- 172.21.15.84:0/3836937953 learned_addr learned my addr 172.21.15.84:0/3836937953 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-29T11:28:38.951 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:38.957+0000 7f544548b640 1 -- 172.21.15.84:0/3836937953 <== mon.1 v1:172.21.15.179:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2798080509 0 0) 0x7f5438036430 con 0x7f5438038dc0 2024-03-29T11:28:38.951 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:38.957+0000 7f544548b640 10 monclient(hunting): _negotiate 2024-03-29T11:28:38.951 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:38.957+0000 7f544548b640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:28:38.951 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:38.957+0000 7f544548b640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:28:38.951 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:38.957+0000 7f544548b640 10 monclient(hunting): my global_id is 4489 2024-03-29T11:28:38.951 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:38.957+0000 7f544548b640 1 -- 172.21.15.84:0/3836937953 --> v1:172.21.15.179:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f542c003520 con 0x7f5438038dc0 2024-03-29T11:28:38.951 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:38.957+0000 7f544548b640 1 -- 172.21.15.84:0/3836937953 <== mon.0 v1:172.21.15.84:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3776446903 0 0) 0x7f54380c54c0 con 0x7f5438030660 2024-03-29T11:28:38.951 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:38.957+0000 7f544548b640 10 monclient(hunting): _negotiate 2024-03-29T11:28:38.951 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:38.957+0000 7f544548b640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:28:38.951 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:38.957+0000 7f544548b640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:28:38.951 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:38.957+0000 7f544548b640 10 monclient(hunting): my global_id is 4542 2024-03-29T11:28:38.951 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:38.957+0000 7f544548b640 1 -- 172.21.15.84:0/3836937953 --> v1:172.21.15.84:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5438036430 con 0x7f5438030660 2024-03-29T11:28:38.952 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:38.958+0000 7f544548b640 1 -- 172.21.15.84:0/3836937953 <== mon.1 v1:172.21.15.179:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1677705972 0 0) 0x7f542c003520 con 0x7f5438038dc0 2024-03-29T11:28:38.952 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:38.958+0000 7f544548b640 1 -- 172.21.15.84:0/3836937953 >> v1:172.21.15.84:6790/0 conn(0x7f54380389e0 legacy=0x7f5438033520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:38.952 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:38.958+0000 7f544548b640 1 -- 172.21.15.84:0/3836937953 >> v1:172.21.15.84:6789/0 conn(0x7f5438030660 legacy=0x7f54380d3170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:38.952 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:38.958+0000 7f544548b640 10 monclient: _finish_hunting 0 2024-03-29T11:28:38.952 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:38.958+0000 7f544548b640 1 monclient: found mon.noname-c 2024-03-29T11:28:38.952 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:38.958+0000 7f544548b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:28:38.952 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:38.958+0000 7f544548b640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.179:6789/0 2024-03-29T11:28:38.952 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:38.958+0000 7f544548b640 1 -- 172.21.15.84:0/3836937953 --> v1:172.21.15.179:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5438035690 con 0x7f5438038dc0 2024-03-29T11:28:38.952 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:38.958+0000 7f544548b640 10 monclient: _finish_auth 0 2024-03-29T11:28:38.952 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:38.958+0000 7f544548b640 10 monclient: _check_auth_tickets 2024-03-29T11:28:38.952 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:38.958+0000 7f544548b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:28:38.952 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:38.958+0000 7f544548b640 1 -- 172.21.15.84:0/3836937953 <== mon.1 v1:172.21.15.179:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4236868694 0 0) 0x7f5430001690 con 0x7f5438038dc0 2024-03-29T11:28:38.952 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:38.958+0000 7f544548b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:28:38.952 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:38.958+0000 7f544548b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:28:38.952 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:38.958+0000 7f544548b640 10 monclient: dump: 2024-03-29T11:28:38.953 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T11:28:38.953 INFO:teuthology.misc.health.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:28:38.953 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T11:18:53.864025+0000 2024-03-29T11:28:38.953 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T11:18:53.864025+0000 2024-03-29T11:28:38.953 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T11:28:38.953 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:28:38.953 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T11:28:38.954 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T11:28:38.954 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T11:28:38.954 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:28:38.954 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:38.958+0000 7f544748f640 5 monclient: authenticate success, global_id 4489 2024-03-29T11:28:38.954 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:38.958+0000 7f544748f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:28:38.954 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:38.958+0000 7f544548b640 1 -- 172.21.15.84:0/3836937953 <== mon.1 v1:172.21.15.179:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5430001920 con 0x7f5438038dc0 2024-03-29T11:28:38.954 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:38.958+0000 7f544548b640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:28:38.954 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:38.958+0000 7f544548b640 1 -- 172.21.15.84:0/3836937953 <== mon.1 v1:172.21.15.179:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4236868694 0 0) 0x7f5430001cf0 con 0x7f5438038dc0 2024-03-29T11:28:38.954 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:38.958+0000 7f544548b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:28:38.955 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:38.958+0000 7f544548b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:28:38.955 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:38.958+0000 7f544548b640 10 monclient: dump: 2024-03-29T11:28:38.955 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T11:28:38.955 INFO:teuthology.misc.health.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:28:38.955 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T11:18:53.864025+0000 2024-03-29T11:28:38.955 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T11:18:53.864025+0000 2024-03-29T11:28:38.955 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T11:28:38.955 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:28:38.955 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T11:28:38.955 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T11:28:38.955 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T11:28:38.955 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:28:38.955 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:38.958+0000 7f544748f640 10 monclient: get_monmap_and_config success 2024-03-29T11:28:38.955 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:38.958+0000 7f544748f640 4 set_mon_vals no callback set 2024-03-29T11:28:38.955 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:38.959+0000 7f544748f640 10 monclient: shutdown 2024-03-29T11:28:38.956 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:38.959+0000 7f544748f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:28:38.956 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:38.959+0000 7f544748f640 1 -- 172.21.15.84:0/3836937953 >> v1:172.21.15.179:6789/0 conn(0x7f5438038dc0 legacy=0x7f54380cadd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:38.956 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:38.960+0000 7f544748f640 1 -- 172.21.15.84:0/3836937953 shutdown_connections 2024-03-29T11:28:38.956 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:38.960+0000 7f544748f640 1 -- 172.21.15.84:0/3836937953 >> 172.21.15.84:0/3836937953 conn(0x7f5438005c00 msgr2=0x7f5438006010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:28:38.956 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:38.960+0000 7f544748f640 1 -- 172.21.15.84:0/3836937953 shutdown_connections 2024-03-29T11:28:38.956 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:38.960+0000 7f544748f640 1 -- 172.21.15.84:0/3836937953 wait complete. 2024-03-29T11:28:38.962 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:38.961+0000 7f544748f640 10 monclient: build_initial_monmap 2024-03-29T11:28:38.962 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:38.961+0000 7f544748f640 10 monclient: monmap: 2024-03-29T11:28:38.962 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-29T11:28:38.962 INFO:teuthology.misc.health.smithi084.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:28:38.963 INFO:teuthology.misc.health.smithi084.stderr:last_changed 0.000000 2024-03-29T11:28:38.963 INFO:teuthology.misc.health.smithi084.stderr:created 0.000000 2024-03-29T11:28:38.963 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-29T11:28:38.963 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:28:38.963 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-29T11:28:38.963 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.noname-b 2024-03-29T11:28:38.963 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-c 2024-03-29T11:28:38.963 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:28:38.963 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:38.962+0000 7f544748f640 1 librados: starting msgr at 2024-03-29T11:28:38.963 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:38.962+0000 7f544748f640 1 librados: starting objecter 2024-03-29T11:28:38.963 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:38.962+0000 7f544748f640 1 Processor -- start 2024-03-29T11:28:38.963 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:38.962+0000 7f544748f640 1 -- start start 2024-03-29T11:28:38.963 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:38.962+0000 7f544748f640 1 librados: setting wanted keys 2024-03-29T11:28:38.963 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:38.962+0000 7f544748f640 1 librados: calling monclient init 2024-03-29T11:28:38.964 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:38.962+0000 7f544748f640 10 monclient: init 2024-03-29T11:28:38.964 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:38.963+0000 7f544748f640 10 monclient: _reopen_session rank -1 2024-03-29T11:28:38.964 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:38.963+0000 7f544748f640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:28:38.964 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:38.964+0000 7f544748f640 10 monclient(hunting): picked mon.noname-b con 0x7f5438030660 addr v1:172.21.15.179:6789/0 2024-03-29T11:28:38.964 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:38.964+0000 7f544748f640 10 monclient(hunting): picked mon.noname-a con 0x7f54380389e0 addr v1:172.21.15.84:6789/0 2024-03-29T11:28:38.964 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:38.964+0000 7f544748f640 10 monclient(hunting): picked mon.noname-c con 0x7f54380c58c0 addr v1:172.21.15.84:6790/0 2024-03-29T11:28:38.964 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:38.964+0000 7f544748f640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54380c7480 con 0x7f54380389e0 2024-03-29T11:28:38.964 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:38.964+0000 7f544748f640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54380c76f0 con 0x7f5438030660 2024-03-29T11:28:38.964 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:38.964+0000 7f544748f640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f543802faf0 con 0x7f54380c58c0 2024-03-29T11:28:38.964 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:38.964+0000 7f544748f640 10 monclient(hunting): _renew_subs 2024-03-29T11:28:38.964 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:38.964+0000 7f544748f640 10 monclient(hunting): authenticate will time out at 1380.258674s 2024-03-29T11:28:38.964 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:38.967+0000 7f5446c8e640 1 --1- >> v1:172.21.15.84:6790/0 conn(0x7f54380c58c0 0x7f5438168250 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.84:6790/0 says I am v1:172.21.15.84:46334/0 (socket says 172.21.15.84:46334) 2024-03-29T11:28:38.964 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:38.967+0000 7f5446c8e640 1 -- 172.21.15.84:0/2521304031 learned_addr learned my addr 172.21.15.84:0/2521304031 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-29T11:28:38.964 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:38.967+0000 7f5436ffd640 10 client.?.objecter ms_handle_connect 0x7f54380c58c0 2024-03-29T11:28:38.965 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:38.967+0000 7f5436ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:28:38.965 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:38.967+0000 7f5436ffd640 1 -- 172.21.15.84:0/2521304031 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2468755476 0 0) 0x7f543802faf0 con 0x7f54380c58c0 2024-03-29T11:28:38.965 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:38.967+0000 7f5436ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:28:38.965 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:38.967+0000 7f5436ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:28:38.965 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:38.967+0000 7f5436ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:28:38.965 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:38.967+0000 7f5436ffd640 10 monclient(hunting): my global_id is 4604 2024-03-29T11:28:38.965 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:38.967+0000 7f5436ffd640 1 -- 172.21.15.84:0/2521304031 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5424004eb0 con 0x7f54380c58c0 2024-03-29T11:28:38.965 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:38.968+0000 7f5436ffd640 10 client.?.objecter ms_handle_connect 0x7f5438030660 2024-03-29T11:28:38.965 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:38.968+0000 7f5436ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:28:38.965 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:38.968+0000 7f5436ffd640 1 -- 172.21.15.84:0/2521304031 <== mon.2 v1:172.21.15.84:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1830848593 0 0) 0x7f5424004eb0 con 0x7f54380c58c0 2024-03-29T11:28:38.965 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:38.968+0000 7f5436ffd640 1 -- 172.21.15.84:0/2521304031 --> v1:172.21.15.84:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f543802faf0 con 0x7f54380c58c0 2024-03-29T11:28:38.965 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:38.968+0000 7f5436ffd640 1 -- 172.21.15.84:0/2521304031 <== mon.2 v1:172.21.15.84:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4236868694 0 0) 0x7f544004fdd0 con 0x7f54380c58c0 2024-03-29T11:28:38.965 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:38.968+0000 7f5436ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:28:38.965 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:38.968+0000 7f5436ffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:28:38.966 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:38.968+0000 7f5436ffd640 10 monclient(hunting): dump: 2024-03-29T11:28:38.966 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T11:28:38.966 INFO:teuthology.misc.health.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:28:38.966 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T11:18:53.864025+0000 2024-03-29T11:28:38.966 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T11:18:53.864025+0000 2024-03-29T11:28:38.966 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T11:28:38.966 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:28:38.966 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T11:28:38.966 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T11:28:38.966 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T11:28:38.966 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:28:38.966 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:38.968+0000 7f5436ffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:28:38.966 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:38.968+0000 7f5436ffd640 1 -- 172.21.15.84:0/2521304031 <== mon.2 v1:172.21.15.84:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3852052794 0 0) 0x7f543802faf0 con 0x7f54380c58c0 2024-03-29T11:28:38.966 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:38.968+0000 7f5436ffd640 1 -- 172.21.15.84:0/2521304031 >> v1:172.21.15.179:6789/0 conn(0x7f5438030660 legacy=0x7f543815cd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:38.966 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:38.968+0000 7f5436ffd640 1 -- 172.21.15.84:0/2521304031 >> v1:172.21.15.84:6789/0 conn(0x7f54380389e0 legacy=0x7f54380c7cf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:28:38.967 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:38.968+0000 7f5436ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:28:38.967 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:38.968+0000 7f5436ffd640 1 monclient: found mon.c 2024-03-29T11:28:38.967 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:38.968+0000 7f5436ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:28:38.967 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:38.968+0000 7f5436ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-29T11:28:38.967 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:38.968+0000 7f5436ffd640 1 -- 172.21.15.84:0/2521304031 --> v1:172.21.15.84:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f543802fdd0 con 0x7f54380c58c0 2024-03-29T11:28:38.967 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:38.968+0000 7f5436ffd640 10 monclient: _finish_auth 0 2024-03-29T11:28:38.967 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:38.968+0000 7f5436ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:28:38.967 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:38.968+0000 7f5436ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:28:38.967 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:38.968+0000 7f5436ffd640 1 -- 172.21.15.84:0/2521304031 <== mon.2 v1:172.21.15.84:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f544004f400 con 0x7f54380c58c0 2024-03-29T11:28:38.967 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:38.968+0000 7f5436ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:28:38.967 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:38.968+0000 7f5436ffd640 1 -- 172.21.15.84:0/2521304031 <== mon.2 v1:172.21.15.84:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4236868694 0 0) 0x7f5440075f10 con 0x7f54380c58c0 2024-03-29T11:28:38.967 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:38.968+0000 7f5436ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:28:38.967 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:38.968+0000 7f5436ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:28:38.967 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:38.968+0000 7f5436ffd640 10 monclient: dump: 2024-03-29T11:28:38.968 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T11:28:38.968 INFO:teuthology.misc.health.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:28:38.968 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T11:18:53.864025+0000 2024-03-29T11:28:38.968 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T11:18:53.864025+0000 2024-03-29T11:28:38.968 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T11:28:38.968 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:28:38.968 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T11:28:38.968 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T11:28:38.968 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T11:28:38.968 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:28:38.968 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:38.968+0000 7f5444c8a640 4 set_mon_vals no callback set 2024-03-29T11:28:38.968 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:38.970+0000 7f544748f640 5 monclient: authenticate success, global_id 4604 2024-03-29T11:28:38.968 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:38.970+0000 7f544748f640 10 monclient: _renew_subs 2024-03-29T11:28:38.968 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:38.970+0000 7f544748f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-29T11:28:38.968 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:38.970+0000 7f544748f640 1 -- 172.21.15.84:0/2521304031 --> v1:172.21.15.84:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f543815a470 con 0x7f54380c58c0 2024-03-29T11:28:38.969 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:38.970+0000 7f544748f640 10 client.4604.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:28:38.969 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:38.970+0000 7f544748f640 10 monclient: _renew_subs 2024-03-29T11:28:38.969 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:38.970+0000 7f544748f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-29T11:28:38.969 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:38.970+0000 7f544748f640 1 -- 172.21.15.84:0/2521304031 --> v1:172.21.15.84:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f54380c6700 con 0x7f54380c58c0 2024-03-29T11:28:38.969 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:38.970+0000 7f544748f640 1 librados: init done 2024-03-29T11:28:38.970 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:38.972+0000 7f5436ffd640 1 -- 172.21.15.84:0/2521304031 <== mon.2 v1:172.21.15.84:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4033942909 0 0) 0x7f5440067bf0 con 0x7f54380c58c0 2024-03-29T11:28:38.970 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:38.973+0000 7f5436ffd640 1 -- 172.21.15.84:0/2521304031 <== mon.2 v1:172.21.15.84:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2808139587 0 0) 0x7f54400ff610 con 0x7f54380c58c0 2024-03-29T11:28:38.970 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:38.973+0000 7f5436ffd640 10 client.4604.objecter ms_dispatch 0x7f5440000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:28:38.971 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:38.973+0000 7f5436ffd640 3 client.4604.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:28:38.971 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:38.973+0000 7f5436ffd640 3 client.4604.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:28:38.971 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:38.973+0000 7f5436ffd640 20 client.4604.objecter dump_active .. 0 homeless 2024-03-29T11:28:38.971 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:38.973+0000 7f544748f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:28:38.971 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:38.973+0000 7f544748f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:28:38.971 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:38.973+0000 7f544748f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-29T11:28:38.971 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:38.973+0000 7f544748f640 1 -- 172.21.15.84:0/2521304031 --> v1:172.21.15.84:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5438007b70 con 0x7f54380c58c0 2024-03-29T11:28:38.971 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:38.975+0000 7f544648d640 10 monclient: get_auth_request con 0x7f542406db20 auth_method 0 2024-03-29T11:28:38.971 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:38.975+0000 7f5436ffd640 10 client.4604.objecter ms_handle_connect 0x7f542406db20 2024-03-29T11:28:38.973 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:38.983+0000 7f5436ffd640 1 -- 172.21.15.84:0/2521304031 <== mon.2 v1:172.21.15.84:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f54400caf20 con 0x7f54380c58c0 2024-03-29T11:28:38.974 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:38.983+0000 7f5436ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:28:38.974 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:38.983+0000 7f5436ffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:28:39.306 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:39.315+0000 7f544748f640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:28:39.306 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:39.315+0000 7f544748f640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:28:39.306 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:39.315+0000 7f544748f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-29T11:28:39.306 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:39.315+0000 7f544748f640 1 -- 172.21.15.84:0/2521304031 --> v1:172.21.15.84:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f543803c3b0 con 0x7f54380c58c0 2024-03-29T11:28:39.307 INFO:teuthology.misc.health.smithi084.stdout: 2024-03-29T11:28:39.307 INFO:teuthology.misc.health.smithi084.stdout:{"epoch":10,"fsid":"2c3e002f-9eb0-4860-853b-4ef1ef562213","created":"2024-03-29T11:19:32.029169+0000","modified":"2024-03-29T11:19:34.200299+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:34.200299+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"5c39e276-2034-4716-bbc0-57a634eefa92","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"cd580e99-109f-4bf1-88c0-a5987bc8b9ed","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"11e60a5a-051d-453f-a190-66387cb35592","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"2c329fbe-bbc4-4030-bf87-d711cad04679","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f4213d4a-bd36-4433-91ae-d7e2b02481d3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"f0cf44ea-6055-48af-805b-aed858102ff9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"d895eaa0-cc0b-4ee4-9b5a-2dfcd89a60b7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"9bc5a7bc-8f52-4db9-96d6-de5e68d52eaa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:28:39.307 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:39.316+0000 7f5436ffd640 1 -- 172.21.15.84:0/2521304031 <== mon.2 v1:172.21.15.84:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4209477549) 0x7f54400cce50 con 0x7f54380c58c0 2024-03-29T11:28:39.307 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:39.316+0000 7f5436ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:28:39.307 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:39.316+0000 7f5436ffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:28:39.311 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:39.320+0000 7f544748f640 10 librados: watch_flush enter 2024-03-29T11:28:39.311 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:39.320+0000 7f544748f640 10 librados: watch_flush exit 2024-03-29T11:28:39.311 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:39.320+0000 7f544748f640 20 client.4604.objecter shutdown clearing up homeless session... 2024-03-29T11:28:39.311 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:39.320+0000 7f544748f640 10 client.4604.objecter successfully canceled tick 2024-03-29T11:28:39.311 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:39.320+0000 7f544748f640 1 -- 172.21.15.84:0/2521304031 >> v1:172.21.15.179:6816/2981134980 conn(0x7f542406db20 legacy=0x7f542406ffe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:39.311 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:39.320+0000 7f544748f640 10 monclient: shutdown 2024-03-29T11:28:39.311 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:39.320+0000 7f544748f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:28:39.311 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:39.320+0000 7f544748f640 1 -- 172.21.15.84:0/2521304031 >> v1:172.21.15.84:6790/0 conn(0x7f54380c58c0 legacy=0x7f5438168250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:39.311 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:39.320+0000 7f544748f640 1 -- 172.21.15.84:0/2521304031 shutdown_connections 2024-03-29T11:28:39.311 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:39.320+0000 7f544748f640 1 -- 172.21.15.84:0/2521304031 >> 172.21.15.84:0/2521304031 conn(0x7f5438005c00 msgr2=0x7f5438034090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:28:39.312 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:39.321+0000 7f544748f640 1 -- 172.21.15.84:0/2521304031 shutdown_connections 2024-03-29T11:28:39.312 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:39.321+0000 7f544748f640 1 -- 172.21.15.84:0/2521304031 wait complete. 2024-03-29T11:28:39.312 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:39.321+0000 7f544748f640 1 librados: shutdown 2024-03-29T11:28:39.326 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:28:46.134 DEBUG:teuthology.orchestra.run.smithi084:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:28:46.284 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:46.294+0000 7fb65c530640 10 monclient: get_monmap_and_config 2024-03-29T11:28:46.284 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:46.294+0000 7fb65c530640 10 monclient: build_initial_monmap 2024-03-29T11:28:46.284 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:46.294+0000 7fb65c530640 10 monclient: monmap: 2024-03-29T11:28:46.284 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-29T11:28:46.284 INFO:teuthology.misc.health.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:28:46.285 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T11:28:46.294981+0000 2024-03-29T11:28:46.285 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T11:28:46.294981+0000 2024-03-29T11:28:46.285 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-29T11:28:46.285 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:28:46.285 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-29T11:28:46.285 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.noname-c 2024-03-29T11:28:46.285 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-b 2024-03-29T11:28:46.286 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:28:46.289 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:46.297+0000 7fb65c530640 1 Processor -- start 2024-03-29T11:28:46.289 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:46.297+0000 7fb65c530640 1 -- start start 2024-03-29T11:28:46.289 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:46.297+0000 7fb65c530640 10 monclient: init 2024-03-29T11:28:46.291 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:46.299+0000 7fb65c530640 10 monclient: _reopen_session rank -1 2024-03-29T11:28:46.291 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:46.299+0000 7fb65c530640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:28:46.292 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:46.299+0000 7fb65c530640 10 monclient(hunting): picked mon.noname-b con 0x7fb6540b4890 addr v1:172.21.15.84:6790/0 2024-03-29T11:28:46.292 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:46.299+0000 7fb65c530640 10 monclient(hunting): picked mon.noname-a con 0x7fb65413c040 addr v1:172.21.15.84:6789/0 2024-03-29T11:28:46.292 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:46.299+0000 7fb65c530640 10 monclient(hunting): picked mon.noname-c con 0x7fb6541427e0 addr v1:172.21.15.179:6789/0 2024-03-29T11:28:46.292 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:46.299+0000 7fb65c530640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb6540b42f0 con 0x7fb65413c040 2024-03-29T11:28:46.292 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:46.299+0000 7fb65c530640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb6540aa7a0 con 0x7fb6541427e0 2024-03-29T11:28:46.292 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:46.299+0000 7fb65c530640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb6540aaa10 con 0x7fb6540b4890 2024-03-29T11:28:46.292 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:46.299+0000 7fb65c530640 10 monclient(hunting): _renew_subs 2024-03-29T11:28:46.292 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:46.299+0000 7fb65c530640 10 monclient(hunting): authenticate will time out at 1387.593851s 2024-03-29T11:28:46.293 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:46.303+0000 7fb65a2cc640 1 --1- >> v1:172.21.15.84:6790/0 conn(0x7fb6540b4890 0x7fb65413b930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.84:6790/0 says I am v1:172.21.15.84:52178/0 (socket says 172.21.15.84:52178) 2024-03-29T11:28:46.293 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:46.303+0000 7fb65a2cc640 1 -- 172.21.15.84:0/2934598571 learned_addr learned my addr 172.21.15.84:0/2934598571 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-29T11:28:46.317 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:46.303+0000 7fb6592ca640 1 -- 172.21.15.84:0/2934598571 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2947623448 0 0) 0x7fb6540aaa10 con 0x7fb6540b4890 2024-03-29T11:28:46.317 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:46.303+0000 7fb6592ca640 10 monclient(hunting): _negotiate 2024-03-29T11:28:46.318 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:46.303+0000 7fb6592ca640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:28:46.318 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:46.303+0000 7fb6592ca640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:28:46.318 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:46.303+0000 7fb6592ca640 10 monclient(hunting): my global_id is 4607 2024-03-29T11:28:46.318 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:46.303+0000 7fb6592ca640 1 -- 172.21.15.84:0/2934598571 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb640003520 con 0x7fb6540b4890 2024-03-29T11:28:46.318 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:46.304+0000 7fb6592ca640 1 -- 172.21.15.84:0/2934598571 <== mon.0 v1:172.21.15.84:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1071094577 0 0) 0x7fb6540b42f0 con 0x7fb65413c040 2024-03-29T11:28:46.318 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:46.304+0000 7fb6592ca640 10 monclient(hunting): _negotiate 2024-03-29T11:28:46.318 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:46.304+0000 7fb6592ca640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:28:46.318 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:46.304+0000 7fb6592ca640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:28:46.318 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:46.304+0000 7fb6592ca640 10 monclient(hunting): my global_id is 4545 2024-03-29T11:28:46.319 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:46.304+0000 7fb6592ca640 1 -- 172.21.15.84:0/2934598571 --> v1:172.21.15.84:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb640003840 con 0x7fb65413c040 2024-03-29T11:28:46.319 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:46.304+0000 7fb6592ca640 1 -- 172.21.15.84:0/2934598571 <== mon.1 v1:172.21.15.179:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1995436138 0 0) 0x7fb6540aa7a0 con 0x7fb6541427e0 2024-03-29T11:28:46.319 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:46.304+0000 7fb6592ca640 10 monclient(hunting): _negotiate 2024-03-29T11:28:46.319 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:46.304+0000 7fb6592ca640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:28:46.319 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:46.304+0000 7fb6592ca640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:28:46.319 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:46.304+0000 7fb6592ca640 10 monclient(hunting): my global_id is 4495 2024-03-29T11:28:46.319 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:46.304+0000 7fb6592ca640 1 -- 172.21.15.84:0/2934598571 --> v1:172.21.15.179:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb6540b42f0 con 0x7fb6541427e0 2024-03-29T11:28:46.319 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:46.304+0000 7fb6592ca640 1 -- 172.21.15.84:0/2934598571 <== mon.2 v1:172.21.15.84:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2186359712 0 0) 0x7fb640003520 con 0x7fb6540b4890 2024-03-29T11:28:46.319 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:46.304+0000 7fb6592ca640 1 -- 172.21.15.84:0/2934598571 >> v1:172.21.15.179:6789/0 conn(0x7fb6541427e0 legacy=0x7fb65414d880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:46.319 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:46.304+0000 7fb6592ca640 1 -- 172.21.15.84:0/2934598571 >> v1:172.21.15.84:6789/0 conn(0x7fb65413c040 legacy=0x7fb6541420d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:46.319 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:46.304+0000 7fb6592ca640 10 monclient: _finish_hunting 0 2024-03-29T11:28:46.319 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:46.304+0000 7fb6592ca640 1 monclient: found mon.noname-b 2024-03-29T11:28:46.319 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:46.304+0000 7fb6592ca640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:28:46.320 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:46.304+0000 7fb6592ca640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.84:6790/0 2024-03-29T11:28:46.320 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:46.304+0000 7fb6592ca640 1 -- 172.21.15.84:0/2934598571 --> v1:172.21.15.84:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6540aacf0 con 0x7fb6540b4890 2024-03-29T11:28:46.320 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:46.304+0000 7fb6592ca640 10 monclient: _finish_auth 0 2024-03-29T11:28:46.320 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:46.304+0000 7fb6592ca640 10 monclient: _check_auth_tickets 2024-03-29T11:28:46.320 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:46.304+0000 7fb6592ca640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:28:46.320 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:46.304+0000 7fb6592ca640 1 -- 172.21.15.84:0/2934598571 <== mon.2 v1:172.21.15.84:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4236868694 0 0) 0x7fb64c0019f0 con 0x7fb6540b4890 2024-03-29T11:28:46.320 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:46.304+0000 7fb6592ca640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:28:46.320 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:46.304+0000 7fb6592ca640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:28:46.320 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:46.304+0000 7fb6592ca640 10 monclient: dump: 2024-03-29T11:28:46.320 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T11:28:46.320 INFO:teuthology.misc.health.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:28:46.320 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T11:18:53.864025+0000 2024-03-29T11:28:46.320 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T11:18:53.864025+0000 2024-03-29T11:28:46.321 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T11:28:46.321 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:28:46.321 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T11:28:46.321 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T11:28:46.321 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T11:28:46.321 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:28:46.321 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:46.307+0000 7fb65c530640 5 monclient: authenticate success, global_id 4607 2024-03-29T11:28:46.322 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:46.307+0000 7fb65c530640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:28:46.322 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:46.308+0000 7fb6592ca640 1 -- 172.21.15.84:0/2934598571 <== mon.2 v1:172.21.15.84:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb64c001b50 con 0x7fb6540b4890 2024-03-29T11:28:46.322 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:46.308+0000 7fb6592ca640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:28:46.323 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:46.308+0000 7fb6592ca640 1 -- 172.21.15.84:0/2934598571 <== mon.2 v1:172.21.15.84:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4236868694 0 0) 0x7fb64c001ee0 con 0x7fb6540b4890 2024-03-29T11:28:46.323 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:46.308+0000 7fb6592ca640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:28:46.323 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:46.308+0000 7fb6592ca640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:28:46.323 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:46.308+0000 7fb6592ca640 10 monclient: dump: 2024-03-29T11:28:46.323 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T11:28:46.323 INFO:teuthology.misc.health.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:28:46.324 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T11:18:53.864025+0000 2024-03-29T11:28:46.324 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T11:18:53.864025+0000 2024-03-29T11:28:46.324 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T11:28:46.324 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:28:46.324 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T11:28:46.324 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T11:28:46.325 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T11:28:46.325 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:28:46.325 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:46.308+0000 7fb65c530640 10 monclient: get_monmap_and_config success 2024-03-29T11:28:46.325 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:46.308+0000 7fb65c530640 4 set_mon_vals no callback set 2024-03-29T11:28:46.325 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:46.308+0000 7fb65c530640 10 monclient: shutdown 2024-03-29T11:28:46.325 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:46.309+0000 7fb65c530640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:28:46.325 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:46.309+0000 7fb65c530640 1 -- 172.21.15.84:0/2934598571 >> v1:172.21.15.84:6790/0 conn(0x7fb6540b4890 legacy=0x7fb65413b930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:46.326 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:46.309+0000 7fb65c530640 1 -- 172.21.15.84:0/2934598571 shutdown_connections 2024-03-29T11:28:46.326 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:46.309+0000 7fb65c530640 1 -- 172.21.15.84:0/2934598571 >> 172.21.15.84:0/2934598571 conn(0x7fb65406d180 msgr2=0x7fb654066a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:28:46.326 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:46.312+0000 7fb65c530640 1 -- 172.21.15.84:0/2934598571 shutdown_connections 2024-03-29T11:28:46.326 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:46.312+0000 7fb65c530640 1 -- 172.21.15.84:0/2934598571 wait complete. 2024-03-29T11:28:46.326 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:46.313+0000 7fb65c530640 10 monclient: build_initial_monmap 2024-03-29T11:28:46.327 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:46.313+0000 7fb65c530640 10 monclient: monmap: 2024-03-29T11:28:46.327 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-29T11:28:46.327 INFO:teuthology.misc.health.smithi084.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:28:46.327 INFO:teuthology.misc.health.smithi084.stderr:last_changed 0.000000 2024-03-29T11:28:46.327 INFO:teuthology.misc.health.smithi084.stderr:created 0.000000 2024-03-29T11:28:46.327 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-29T11:28:46.327 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:28:46.328 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-29T11:28:46.328 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.noname-b 2024-03-29T11:28:46.328 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-c 2024-03-29T11:28:46.328 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:28:46.328 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:46.314+0000 7fb65c530640 1 librados: starting msgr at 2024-03-29T11:28:46.328 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:46.314+0000 7fb65c530640 1 librados: starting objecter 2024-03-29T11:28:46.329 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:46.314+0000 7fb65c530640 1 Processor -- start 2024-03-29T11:28:46.329 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:46.314+0000 7fb65c530640 1 -- start start 2024-03-29T11:28:46.329 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:46.314+0000 7fb65c530640 1 librados: setting wanted keys 2024-03-29T11:28:46.329 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:46.314+0000 7fb65c530640 1 librados: calling monclient init 2024-03-29T11:28:46.329 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:46.314+0000 7fb65c530640 10 monclient: init 2024-03-29T11:28:46.329 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:46.315+0000 7fb65c530640 10 monclient: _reopen_session rank -1 2024-03-29T11:28:46.329 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:46.315+0000 7fb65c530640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:28:46.330 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:46.315+0000 7fb65c530640 10 monclient(hunting): picked mon.noname-b con 0x7fb6540b4890 addr v1:172.21.15.179:6789/0 2024-03-29T11:28:46.330 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:46.316+0000 7fb65c530640 10 monclient(hunting): picked mon.noname-c con 0x7fb65413c040 addr v1:172.21.15.84:6790/0 2024-03-29T11:28:46.330 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:46.317+0000 7fb65c530640 10 monclient(hunting): picked mon.noname-a con 0x7fb6541427e0 addr v1:172.21.15.84:6789/0 2024-03-29T11:28:46.330 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:46.317+0000 7fb65c530640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb654071570 con 0x7fb6541427e0 2024-03-29T11:28:46.330 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:46.317+0000 7fb65c530640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb6540abdf0 con 0x7fb6540b4890 2024-03-29T11:28:46.330 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:46.317+0000 7fb65c530640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb6540ac0d0 con 0x7fb65413c040 2024-03-29T11:28:46.331 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:46.317+0000 7fb65c530640 10 monclient(hunting): _renew_subs 2024-03-29T11:28:46.331 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:46.317+0000 7fb65c530640 10 monclient(hunting): authenticate will time out at 1387.611481s 2024-03-29T11:28:46.331 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:46.317+0000 7fb659acb640 1 --1- >> v1:172.21.15.84:6790/0 conn(0x7fb65413c040 0x7fb65406de90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.84:6790/0 says I am v1:172.21.15.84:55248/0 (socket says 172.21.15.84:55248) 2024-03-29T11:28:46.331 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:46.317+0000 7fb659acb640 1 -- 172.21.15.84:0/520396391 learned_addr learned my addr 172.21.15.84:0/520396391 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-29T11:28:46.331 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:46.317+0000 7fb64affd640 10 client.?.objecter ms_handle_connect 0x7fb65413c040 2024-03-29T11:28:46.331 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:46.317+0000 7fb64affd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:28:46.332 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:46.317+0000 7fb64affd640 1 -- 172.21.15.84:0/520396391 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4129451773 0 0) 0x7fb6540ac0d0 con 0x7fb65413c040 2024-03-29T11:28:46.332 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:46.317+0000 7fb64affd640 10 monclient(hunting): _negotiate 2024-03-29T11:28:46.332 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:46.317+0000 7fb64affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:28:46.332 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:46.317+0000 7fb64affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:28:46.332 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:46.317+0000 7fb64affd640 10 monclient(hunting): my global_id is 4610 2024-03-29T11:28:46.332 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:46.317+0000 7fb64affd640 1 -- 172.21.15.84:0/520396391 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb638004eb0 con 0x7fb65413c040 2024-03-29T11:28:46.333 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:46.319+0000 7fb64affd640 10 client.?.objecter ms_handle_connect 0x7fb6541427e0 2024-03-29T11:28:46.333 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:46.319+0000 7fb64affd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:28:46.333 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:46.319+0000 7fb64affd640 1 -- 172.21.15.84:0/520396391 <== mon.0 v1:172.21.15.84:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3401892610 0 0) 0x7fb654071570 con 0x7fb6541427e0 2024-03-29T11:28:46.333 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:46.319+0000 7fb64affd640 10 monclient(hunting): _negotiate 2024-03-29T11:28:46.333 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:46.319+0000 7fb64affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:28:46.333 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:46.319+0000 7fb64affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:28:46.333 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:46.319+0000 7fb64affd640 10 monclient(hunting): my global_id is 4548 2024-03-29T11:28:46.334 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:46.319+0000 7fb64affd640 1 -- 172.21.15.84:0/520396391 --> v1:172.21.15.84:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb6540ac0d0 con 0x7fb6541427e0 2024-03-29T11:28:46.334 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:46.319+0000 7fb64affd640 1 -- 172.21.15.84:0/520396391 <== mon.0 v1:172.21.15.84:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 762548157 0 0) 0x7fb6540ac0d0 con 0x7fb6541427e0 2024-03-29T11:28:46.334 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:46.319+0000 7fb64affd640 1 -- 172.21.15.84:0/520396391 --> v1:172.21.15.84:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb654071570 con 0x7fb6541427e0 2024-03-29T11:28:46.334 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:46.319+0000 7fb64affd640 1 -- 172.21.15.84:0/520396391 <== mon.0 v1:172.21.15.84:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4236868694 0 0) 0x7fb6500018b0 con 0x7fb6541427e0 2024-03-29T11:28:46.334 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:46.319+0000 7fb64affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:28:46.334 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:46.319+0000 7fb64affd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:28:46.334 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:46.319+0000 7fb64affd640 10 monclient(hunting): dump: 2024-03-29T11:28:46.334 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T11:28:46.335 INFO:teuthology.misc.health.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:28:46.335 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T11:18:53.864025+0000 2024-03-29T11:28:46.335 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T11:18:53.864025+0000 2024-03-29T11:28:46.335 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T11:28:46.335 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:28:46.335 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T11:28:46.335 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T11:28:46.335 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T11:28:46.335 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:28:46.336 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:46.319+0000 7fb64affd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:28:46.336 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:46.320+0000 7fb64affd640 1 -- 172.21.15.84:0/520396391 <== mon.0 v1:172.21.15.84:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2875010384 0 0) 0x7fb654071570 con 0x7fb6541427e0 2024-03-29T11:28:46.336 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:46.320+0000 7fb64affd640 1 -- 172.21.15.84:0/520396391 >> v1:172.21.15.84:6790/0 conn(0x7fb65413c040 legacy=0x7fb65406de90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:46.336 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:46.320+0000 7fb64affd640 1 -- 172.21.15.84:0/520396391 >> v1:172.21.15.179:6789/0 conn(0x7fb6540b4890 legacy=0x7fb654071a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:46.336 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:46.321+0000 7fb64affd640 10 monclient: _finish_hunting 0 2024-03-29T11:28:46.336 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:46.321+0000 7fb64affd640 1 monclient: found mon.a 2024-03-29T11:28:46.336 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:46.321+0000 7fb64affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:28:46.336 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:46.321+0000 7fb64affd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.84:6789/0 2024-03-29T11:28:46.336 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:46.321+0000 7fb64affd640 1 -- 172.21.15.84:0/520396391 --> v1:172.21.15.84:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb65406e5a0 con 0x7fb6541427e0 2024-03-29T11:28:46.336 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:46.321+0000 7fb64affd640 10 monclient: _finish_auth 0 2024-03-29T11:28:46.336 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:46.321+0000 7fb64affd640 10 monclient: _check_auth_tickets 2024-03-29T11:28:46.336 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:46.321+0000 7fb64affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:28:46.336 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:46.321+0000 7fb65c530640 5 monclient: authenticate success, global_id 4548 2024-03-29T11:28:46.336 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:46.321+0000 7fb65c530640 10 monclient: _renew_subs 2024-03-29T11:28:46.336 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:46.321+0000 7fb65c530640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.84:6789/0 2024-03-29T11:28:46.337 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:46.321+0000 7fb65c530640 1 -- 172.21.15.84:0/520396391 --> v1:172.21.15.84:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb654071050 con 0x7fb6541427e0 2024-03-29T11:28:46.337 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:46.321+0000 7fb65c530640 10 client.4548.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:28:46.337 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:46.321+0000 7fb65c530640 10 monclient: _renew_subs 2024-03-29T11:28:46.337 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:46.321+0000 7fb65c530640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.84:6789/0 2024-03-29T11:28:46.337 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:46.321+0000 7fb65c530640 1 -- 172.21.15.84:0/520396391 --> v1:172.21.15.84:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb654070c30 con 0x7fb6541427e0 2024-03-29T11:28:46.337 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:46.321+0000 7fb65c530640 1 librados: init done 2024-03-29T11:28:46.337 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:46.322+0000 7fb64affd640 1 -- 172.21.15.84:0/520396391 <== mon.0 v1:172.21.15.84:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb6500035f0 con 0x7fb6541427e0 2024-03-29T11:28:46.337 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:46.322+0000 7fb64affd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:28:46.337 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:46.322+0000 7fb64affd640 1 -- 172.21.15.84:0/520396391 <== mon.0 v1:172.21.15.84:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4236868694 0 0) 0x7fb650003900 con 0x7fb6541427e0 2024-03-29T11:28:46.337 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:46.322+0000 7fb64affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:28:46.337 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:46.322+0000 7fb64affd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:28:46.337 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:46.322+0000 7fb64affd640 10 monclient: dump: 2024-03-29T11:28:46.337 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T11:28:46.337 INFO:teuthology.misc.health.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:28:46.337 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T11:18:53.864025+0000 2024-03-29T11:28:46.338 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T11:18:53.864025+0000 2024-03-29T11:28:46.338 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T11:28:46.338 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:28:46.338 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T11:28:46.338 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T11:28:46.338 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T11:28:46.338 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:28:46.338 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:46.322+0000 7fb658ac9640 4 set_mon_vals no callback set 2024-03-29T11:28:46.338 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:46.322+0000 7fb65c530640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:28:46.338 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:46.323+0000 7fb64affd640 1 -- 172.21.15.84:0/520396391 <== mon.0 v1:172.21.15.84:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4033942909 0 0) 0x7fb6500198d0 con 0x7fb6541427e0 2024-03-29T11:28:46.338 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:46.324+0000 7fb64affd640 1 -- 172.21.15.84:0/520396391 <== mon.0 v1:172.21.15.84:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2808139587 0 0) 0x7fb6500836e0 con 0x7fb6541427e0 2024-03-29T11:28:46.338 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:46.324+0000 7fb64affd640 10 client.4548.objecter ms_dispatch 0x7fb654000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:28:46.338 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:46.324+0000 7fb64affd640 3 client.4548.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:28:46.338 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:46.324+0000 7fb64affd640 3 client.4548.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:28:46.339 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:46.325+0000 7fb64affd640 20 client.4548.objecter dump_active .. 0 homeless 2024-03-29T11:28:46.339 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:46.326+0000 7fb65c530640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:28:46.339 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:46.326+0000 7fb65c530640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.84:6789/0 2024-03-29T11:28:46.339 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:46.326+0000 7fb65c530640 1 -- 172.21.15.84:0/520396391 --> v1:172.21.15.84:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb63c005080 con 0x7fb6541427e0 2024-03-29T11:28:46.339 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:46.326+0000 7fb65a2cc640 10 monclient: get_auth_request con 0x7fb63806def0 auth_method 0 2024-03-29T11:28:46.339 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:46.327+0000 7fb64affd640 10 client.4548.objecter ms_handle_connect 0x7fb63806def0 2024-03-29T11:28:46.339 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:46.334+0000 7fb64affd640 1 -- 172.21.15.84:0/520396391 <== mon.0 v1:172.21.15.84:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fb65004f020 con 0x7fb6541427e0 2024-03-29T11:28:46.339 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:46.334+0000 7fb64affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:28:46.340 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:46.334+0000 7fb64affd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:28:46.647 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:46.656+0000 7fb65c530640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:28:46.648 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:46.656+0000 7fb65c530640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:28:46.648 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:46.656+0000 7fb65c530640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.84:6789/0 2024-03-29T11:28:46.648 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:46.656+0000 7fb65c530640 1 -- 172.21.15.84:0/520396391 --> v1:172.21.15.84:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb63c005330 con 0x7fb6541427e0 2024-03-29T11:28:46.648 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:46.658+0000 7fb64affd640 1 -- 172.21.15.84:0/520396391 <== mon.0 v1:172.21.15.84:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4209477549) 0x7fb650082d90 con 0x7fb6541427e0 2024-03-29T11:28:46.648 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:46.658+0000 7fb64affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:28:46.648 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:46.658+0000 7fb64affd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:28:46.649 INFO:teuthology.misc.health.smithi084.stdout: 2024-03-29T11:28:46.649 INFO:teuthology.misc.health.smithi084.stdout:{"epoch":10,"fsid":"2c3e002f-9eb0-4860-853b-4ef1ef562213","created":"2024-03-29T11:19:32.029169+0000","modified":"2024-03-29T11:19:34.200299+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:34.200299+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"5c39e276-2034-4716-bbc0-57a634eefa92","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"cd580e99-109f-4bf1-88c0-a5987bc8b9ed","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"11e60a5a-051d-453f-a190-66387cb35592","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"2c329fbe-bbc4-4030-bf87-d711cad04679","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f4213d4a-bd36-4433-91ae-d7e2b02481d3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"f0cf44ea-6055-48af-805b-aed858102ff9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"d895eaa0-cc0b-4ee4-9b5a-2dfcd89a60b7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"9bc5a7bc-8f52-4db9-96d6-de5e68d52eaa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:28:46.651 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:46.661+0000 7fb65c530640 10 librados: watch_flush enter 2024-03-29T11:28:46.654 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:46.663+0000 7fb65c530640 10 librados: watch_flush exit 2024-03-29T11:28:46.654 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:46.663+0000 7fb65c530640 20 client.4548.objecter shutdown clearing up homeless session... 2024-03-29T11:28:46.654 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:46.663+0000 7fb65c530640 10 client.4548.objecter successfully canceled tick 2024-03-29T11:28:46.654 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:46.663+0000 7fb65c530640 1 -- 172.21.15.84:0/520396391 >> v1:172.21.15.179:6816/2981134980 conn(0x7fb63806def0 legacy=0x7fb6380703b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:46.654 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:46.663+0000 7fb65c530640 10 monclient: shutdown 2024-03-29T11:28:46.654 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:46.663+0000 7fb65c530640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:28:46.654 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:46.663+0000 7fb65c530640 1 -- 172.21.15.84:0/520396391 >> v1:172.21.15.84:6789/0 conn(0x7fb6541427e0 legacy=0x7fb6541dda60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:46.654 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:46.663+0000 7fb65c530640 1 -- 172.21.15.84:0/520396391 shutdown_connections 2024-03-29T11:28:46.654 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:46.663+0000 7fb65c530640 1 -- 172.21.15.84:0/520396391 >> 172.21.15.84:0/520396391 conn(0x7fb65406d180 msgr2=0x7fb6540b3510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:28:46.654 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:46.663+0000 7fb65c530640 1 -- 172.21.15.84:0/520396391 shutdown_connections 2024-03-29T11:28:46.655 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:46.663+0000 7fb65c530640 1 -- 172.21.15.84:0/520396391 wait complete. 2024-03-29T11:28:46.655 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:46.663+0000 7fb65c530640 1 librados: shutdown 2024-03-29T11:28:46.672 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:28:53.480 DEBUG:teuthology.orchestra.run.smithi084:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:28:53.622 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:53.630+0000 7f17ddf3d640 10 monclient: get_monmap_and_config 2024-03-29T11:28:53.622 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:53.630+0000 7f17ddf3d640 10 monclient: build_initial_monmap 2024-03-29T11:28:53.622 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:53.630+0000 7f17ddf3d640 10 monclient: monmap: 2024-03-29T11:28:53.622 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-29T11:28:53.623 INFO:teuthology.misc.health.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:28:53.623 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T11:28:53.631239+0000 2024-03-29T11:28:53.623 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T11:28:53.631239+0000 2024-03-29T11:28:53.623 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-29T11:28:53.623 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:28:53.623 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-29T11:28:53.623 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.noname-c 2024-03-29T11:28:53.623 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-b 2024-03-29T11:28:53.623 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:28:53.623 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:53.631+0000 7f17ddf3d640 1 Processor -- start 2024-03-29T11:28:53.623 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:53.631+0000 7f17ddf3d640 1 -- start start 2024-03-29T11:28:53.623 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:53.631+0000 7f17ddf3d640 10 monclient: init 2024-03-29T11:28:53.627 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:53.632+0000 7f17ddf3d640 10 monclient: _reopen_session rank -1 2024-03-29T11:28:53.627 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:53.632+0000 7f17ddf3d640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:28:53.627 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:53.633+0000 7f17ddf3d640 10 monclient(hunting): picked mon.noname-b con 0x7f17d80b4710 addr v1:172.21.15.84:6790/0 2024-03-29T11:28:53.627 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:53.633+0000 7f17ddf3d640 10 monclient(hunting): picked mon.noname-a con 0x7f17d80b3970 addr v1:172.21.15.84:6789/0 2024-03-29T11:28:53.627 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:53.633+0000 7f17ddf3d640 10 monclient(hunting): picked mon.noname-c con 0x7f17d813f390 addr v1:172.21.15.179:6789/0 2024-03-29T11:28:53.627 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:53.633+0000 7f17ddf3d640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f17d80af020 con 0x7f17d80b3970 2024-03-29T11:28:53.627 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:53.633+0000 7f17ddf3d640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f17d80b0910 con 0x7f17d813f390 2024-03-29T11:28:53.627 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:53.633+0000 7f17ddf3d640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f17d80b0bf0 con 0x7f17d80b4710 2024-03-29T11:28:53.627 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:53.633+0000 7f17d77fe640 1 --1- >> v1:172.21.15.84:6790/0 conn(0x7f17d80b4710 0x7f17d80b1010 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.84:6790/0 says I am v1:172.21.15.84:50690/0 (socket says 172.21.15.84:50690) 2024-03-29T11:28:53.628 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:53.633+0000 7f17d77fe640 1 -- 172.21.15.84:0/4182758388 learned_addr learned my addr 172.21.15.84:0/4182758388 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-29T11:28:53.628 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:53.633+0000 7f17ddf3d640 10 monclient(hunting): _renew_subs 2024-03-29T11:28:53.628 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:53.633+0000 7f17ddf3d640 10 monclient(hunting): authenticate will time out at 1394.927441s 2024-03-29T11:28:53.628 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:53.633+0000 7f17d67fc640 1 -- 172.21.15.84:0/4182758388 <== mon.1 v1:172.21.15.179:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2740017418 0 0) 0x7f17d80b0910 con 0x7f17d813f390 2024-03-29T11:28:53.628 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:53.633+0000 7f17d67fc640 10 monclient(hunting): _negotiate 2024-03-29T11:28:53.628 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:53.633+0000 7f17d67fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:28:53.628 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:53.633+0000 7f17d67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:28:53.628 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:53.633+0000 7f17d67fc640 10 monclient(hunting): my global_id is 4498 2024-03-29T11:28:53.628 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:53.633+0000 7f17d67fc640 1 -- 172.21.15.84:0/4182758388 --> v1:172.21.15.179:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f17c0003520 con 0x7f17d813f390 2024-03-29T11:28:53.628 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:53.634+0000 7f17d67fc640 1 -- 172.21.15.84:0/4182758388 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1851045029 0 0) 0x7f17d80b0bf0 con 0x7f17d80b4710 2024-03-29T11:28:53.628 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:53.634+0000 7f17d67fc640 10 monclient(hunting): _negotiate 2024-03-29T11:28:53.628 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:53.634+0000 7f17d67fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:28:53.628 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:53.634+0000 7f17d67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:28:53.628 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:53.634+0000 7f17d67fc640 10 monclient(hunting): my global_id is 4613 2024-03-29T11:28:53.628 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:53.634+0000 7f17d67fc640 1 -- 172.21.15.84:0/4182758388 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f17d80b0910 con 0x7f17d80b4710 2024-03-29T11:28:53.629 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:53.634+0000 7f17d67fc640 1 -- 172.21.15.84:0/4182758388 <== mon.0 v1:172.21.15.84:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3512477011 0 0) 0x7f17d80af020 con 0x7f17d80b3970 2024-03-29T11:28:53.629 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:53.634+0000 7f17d67fc640 10 monclient(hunting): _negotiate 2024-03-29T11:28:53.629 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:53.634+0000 7f17d67fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:28:53.629 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:53.634+0000 7f17d67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:28:53.629 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:53.634+0000 7f17d67fc640 10 monclient(hunting): my global_id is 4551 2024-03-29T11:28:53.629 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:53.634+0000 7f17d67fc640 1 -- 172.21.15.84:0/4182758388 --> v1:172.21.15.84:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f17d80b0bf0 con 0x7f17d80b3970 2024-03-29T11:28:53.629 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:53.634+0000 7f17d67fc640 1 -- 172.21.15.84:0/4182758388 <== mon.1 v1:172.21.15.179:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2569821109 0 0) 0x7f17c0003520 con 0x7f17d813f390 2024-03-29T11:28:53.629 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:53.634+0000 7f17d67fc640 1 -- 172.21.15.84:0/4182758388 >> v1:172.21.15.84:6790/0 conn(0x7f17d80b4710 legacy=0x7f17d80b1010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:53.629 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:53.634+0000 7f17d67fc640 1 -- 172.21.15.84:0/4182758388 >> v1:172.21.15.84:6789/0 conn(0x7f17d80b3970 legacy=0x7f17d80af4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:53.629 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:53.634+0000 7f17d67fc640 10 monclient: _finish_hunting 0 2024-03-29T11:28:53.629 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:53.634+0000 7f17d67fc640 1 monclient: found mon.noname-c 2024-03-29T11:28:53.629 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:53.634+0000 7f17d67fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:28:53.629 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:53.634+0000 7f17d67fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.179:6789/0 2024-03-29T11:28:53.629 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:53.634+0000 7f17d67fc640 1 -- 172.21.15.84:0/4182758388 --> v1:172.21.15.179:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17d80aeb00 con 0x7f17d813f390 2024-03-29T11:28:53.630 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:53.634+0000 7f17d67fc640 10 monclient: _finish_auth 0 2024-03-29T11:28:53.630 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:53.634+0000 7f17d67fc640 10 monclient: _check_auth_tickets 2024-03-29T11:28:53.630 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:53.634+0000 7f17d67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:28:53.630 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:53.634+0000 7f17d67fc640 1 -- 172.21.15.84:0/4182758388 <== mon.1 v1:172.21.15.179:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4236868694 0 0) 0x7f17d0001940 con 0x7f17d813f390 2024-03-29T11:28:53.630 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:53.634+0000 7f17d67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:28:53.630 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:53.634+0000 7f17d67fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:28:53.630 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:53.634+0000 7f17d67fc640 10 monclient: dump: 2024-03-29T11:28:53.630 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T11:28:53.630 INFO:teuthology.misc.health.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:28:53.630 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T11:18:53.864025+0000 2024-03-29T11:28:53.630 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T11:18:53.864025+0000 2024-03-29T11:28:53.630 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T11:28:53.630 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:28:53.630 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T11:28:53.631 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T11:28:53.631 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T11:28:53.631 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:28:53.631 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:53.634+0000 7f17ddf3d640 5 monclient: authenticate success, global_id 4498 2024-03-29T11:28:53.631 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:53.634+0000 7f17ddf3d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:28:53.631 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:53.635+0000 7f17d67fc640 1 -- 172.21.15.84:0/4182758388 <== mon.1 v1:172.21.15.179:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f17d0001aa0 con 0x7f17d813f390 2024-03-29T11:28:53.631 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:53.635+0000 7f17d67fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:28:53.631 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:53.635+0000 7f17d67fc640 1 -- 172.21.15.84:0/4182758388 <== mon.1 v1:172.21.15.179:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4236868694 0 0) 0x7f17d0001eb0 con 0x7f17d813f390 2024-03-29T11:28:53.632 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:53.635+0000 7f17d67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:28:53.632 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:53.635+0000 7f17d67fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:28:53.632 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:53.635+0000 7f17d67fc640 10 monclient: dump: 2024-03-29T11:28:53.632 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T11:28:53.632 INFO:teuthology.misc.health.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:28:53.632 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T11:18:53.864025+0000 2024-03-29T11:28:53.632 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T11:18:53.864025+0000 2024-03-29T11:28:53.632 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T11:28:53.632 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:28:53.632 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T11:28:53.633 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T11:28:53.633 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T11:28:53.633 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:28:53.633 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:53.635+0000 7f17ddf3d640 10 monclient: get_monmap_and_config success 2024-03-29T11:28:53.633 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:53.635+0000 7f17ddf3d640 4 set_mon_vals no callback set 2024-03-29T11:28:53.633 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:53.636+0000 7f17ddf3d640 10 monclient: shutdown 2024-03-29T11:28:53.633 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:53.636+0000 7f17ddf3d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:28:53.633 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:53.636+0000 7f17ddf3d640 1 -- 172.21.15.84:0/4182758388 >> v1:172.21.15.179:6789/0 conn(0x7f17d813f390 legacy=0x7f17d813f800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:53.633 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:53.636+0000 7f17ddf3d640 1 -- 172.21.15.84:0/4182758388 shutdown_connections 2024-03-29T11:28:53.633 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:53.636+0000 7f17ddf3d640 1 -- 172.21.15.84:0/4182758388 >> 172.21.15.84:0/4182758388 conn(0x7f17d8066a00 msgr2=0x7f17d8066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:28:53.634 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:53.636+0000 7f17ddf3d640 1 -- 172.21.15.84:0/4182758388 shutdown_connections 2024-03-29T11:28:53.634 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:53.636+0000 7f17ddf3d640 1 -- 172.21.15.84:0/4182758388 wait complete. 2024-03-29T11:28:53.641 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:53.637+0000 7f17ddf3d640 10 monclient: build_initial_monmap 2024-03-29T11:28:53.641 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:53.637+0000 7f17ddf3d640 10 monclient: monmap: 2024-03-29T11:28:53.641 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-29T11:28:53.642 INFO:teuthology.misc.health.smithi084.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:28:53.642 INFO:teuthology.misc.health.smithi084.stderr:last_changed 0.000000 2024-03-29T11:28:53.642 INFO:teuthology.misc.health.smithi084.stderr:created 0.000000 2024-03-29T11:28:53.642 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-29T11:28:53.642 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:28:53.642 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-29T11:28:53.642 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.noname-b 2024-03-29T11:28:53.642 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-c 2024-03-29T11:28:53.642 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:28:53.642 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:53.638+0000 7f17ddf3d640 1 librados: starting msgr at 2024-03-29T11:28:53.642 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:53.638+0000 7f17ddf3d640 1 librados: starting objecter 2024-03-29T11:28:53.642 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:53.638+0000 7f17ddf3d640 1 Processor -- start 2024-03-29T11:28:53.642 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:53.638+0000 7f17ddf3d640 1 -- start start 2024-03-29T11:28:53.642 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:53.638+0000 7f17ddf3d640 1 librados: setting wanted keys 2024-03-29T11:28:53.643 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:53.638+0000 7f17ddf3d640 1 librados: calling monclient init 2024-03-29T11:28:53.643 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:53.638+0000 7f17ddf3d640 10 monclient: init 2024-03-29T11:28:53.643 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:53.639+0000 7f17ddf3d640 10 monclient: _reopen_session rank -1 2024-03-29T11:28:53.643 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:53.639+0000 7f17ddf3d640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:28:53.643 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:53.639+0000 7f17ddf3d640 10 monclient(hunting): picked mon.noname-b con 0x7f17d80b3970 addr v1:172.21.15.179:6789/0 2024-03-29T11:28:53.643 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:53.639+0000 7f17ddf3d640 10 monclient(hunting): picked mon.noname-c con 0x7f17d80b4710 addr v1:172.21.15.84:6790/0 2024-03-29T11:28:53.643 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:53.639+0000 7f17ddf3d640 10 monclient(hunting): picked mon.noname-a con 0x7f17d813f390 addr v1:172.21.15.84:6789/0 2024-03-29T11:28:53.643 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:53.639+0000 7f17ddf3d640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f17d81376a0 con 0x7f17d813f390 2024-03-29T11:28:53.643 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:53.639+0000 7f17ddf3d640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f17d8137980 con 0x7f17d80b3970 2024-03-29T11:28:53.643 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:53.639+0000 7f17ddf3d640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f17d8142580 con 0x7f17d80b4710 2024-03-29T11:28:53.643 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:53.639+0000 7f17ddf3d640 10 monclient(hunting): _renew_subs 2024-03-29T11:28:53.643 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:53.639+0000 7f17ddf3d640 10 monclient(hunting): authenticate will time out at 1394.933812s 2024-03-29T11:28:53.643 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:53.639+0000 7f17d77fe640 1 --1- >> v1:172.21.15.179:6789/0 conn(0x7f17d80b3970 0x7f17d8137e10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.179:6789/0 says I am v1:172.21.15.84:33984/0 (socket says 172.21.15.84:33984) 2024-03-29T11:28:53.644 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:53.639+0000 7f17d77fe640 1 -- 172.21.15.84:0/4069665930 learned_addr learned my addr 172.21.15.84:0/4069665930 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-29T11:28:53.644 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:53.640+0000 7f17bffff640 10 client.?.objecter ms_handle_connect 0x7f17d80b3970 2024-03-29T11:28:53.644 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:53.640+0000 7f17bffff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:28:53.644 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:53.640+0000 7f17bffff640 10 client.?.objecter ms_handle_connect 0x7f17d813f390 2024-03-29T11:28:53.644 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:53.640+0000 7f17bffff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:28:53.644 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:53.640+0000 7f17bffff640 10 client.?.objecter ms_handle_connect 0x7f17d80b4710 2024-03-29T11:28:53.644 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:53.640+0000 7f17bffff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:28:53.644 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:53.640+0000 7f17bffff640 1 -- 172.21.15.84:0/4069665930 <== mon.1 v1:172.21.15.179:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1757114003 0 0) 0x7f17d8137980 con 0x7f17d80b3970 2024-03-29T11:28:53.644 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:53.640+0000 7f17bffff640 10 monclient(hunting): _negotiate 2024-03-29T11:28:53.644 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:53.640+0000 7f17bffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:28:53.644 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:53.640+0000 7f17bffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:28:53.644 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:53.640+0000 7f17bffff640 10 monclient(hunting): my global_id is 4501 2024-03-29T11:28:53.644 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:53.640+0000 7f17bffff640 1 -- 172.21.15.84:0/4069665930 --> v1:172.21.15.179:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f17c4004ee0 con 0x7f17d80b3970 2024-03-29T11:28:53.645 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:53.640+0000 7f17bffff640 1 -- 172.21.15.84:0/4069665930 <== mon.0 v1:172.21.15.84:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 727756501 0 0) 0x7f17d81376a0 con 0x7f17d813f390 2024-03-29T11:28:53.645 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:53.640+0000 7f17bffff640 10 monclient(hunting): _negotiate 2024-03-29T11:28:53.645 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:53.640+0000 7f17bffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:28:53.645 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:53.640+0000 7f17bffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:28:53.645 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:53.640+0000 7f17bffff640 10 monclient(hunting): my global_id is 4554 2024-03-29T11:28:53.645 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:53.640+0000 7f17bffff640 1 -- 172.21.15.84:0/4069665930 --> v1:172.21.15.84:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f17d8137980 con 0x7f17d813f390 2024-03-29T11:28:53.645 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:53.640+0000 7f17bffff640 1 -- 172.21.15.84:0/4069665930 <== mon.0 v1:172.21.15.84:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3828328304 0 0) 0x7f17d8137980 con 0x7f17d813f390 2024-03-29T11:28:53.645 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:53.640+0000 7f17bffff640 1 -- 172.21.15.84:0/4069665930 --> v1:172.21.15.84:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f17d81376a0 con 0x7f17d813f390 2024-03-29T11:28:53.645 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:53.640+0000 7f17bffff640 1 -- 172.21.15.84:0/4069665930 <== mon.1 v1:172.21.15.179:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4110735880 0 0) 0x7f17c4004ee0 con 0x7f17d80b3970 2024-03-29T11:28:53.645 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:53.640+0000 7f17bffff640 1 -- 172.21.15.84:0/4069665930 --> v1:172.21.15.179:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f17d8137980 con 0x7f17d80b3970 2024-03-29T11:28:53.645 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:53.640+0000 7f17bffff640 1 -- 172.21.15.84:0/4069665930 <== mon.1 v1:172.21.15.179:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4236868694 0 0) 0x7f17c8001970 con 0x7f17d80b3970 2024-03-29T11:28:53.645 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:53.640+0000 7f17bffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:28:53.645 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:53.640+0000 7f17bffff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:28:53.645 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:53.640+0000 7f17bffff640 10 monclient(hunting): dump: 2024-03-29T11:28:53.645 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T11:28:53.646 INFO:teuthology.misc.health.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:28:53.646 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T11:18:53.864025+0000 2024-03-29T11:28:53.646 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T11:18:53.864025+0000 2024-03-29T11:28:53.646 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T11:28:53.646 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:28:53.646 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T11:28:53.646 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T11:28:53.646 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T11:28:53.646 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:28:53.646 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:53.640+0000 7f17bffff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:28:53.646 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:53.640+0000 7f17bffff640 1 -- 172.21.15.84:0/4069665930 <== mon.0 v1:172.21.15.84:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4236868694 0 0) 0x7f17d0001eb0 con 0x7f17d813f390 2024-03-29T11:28:53.646 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:53.640+0000 7f17bffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:28:53.646 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:53.640+0000 7f17bffff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:28:53.646 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:53.640+0000 7f17bffff640 10 monclient(hunting): dump: 2024-03-29T11:28:53.646 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T11:28:53.647 INFO:teuthology.misc.health.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:28:53.647 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T11:18:53.864025+0000 2024-03-29T11:28:53.647 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T11:18:53.864025+0000 2024-03-29T11:28:53.647 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T11:28:53.647 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:28:53.647 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T11:28:53.647 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T11:28:53.647 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T11:28:53.647 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:28:53.647 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:53.641+0000 7f17bffff640 1 -- 172.21.15.84:0/4069665930 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4056978642 0 0) 0x7f17d8142580 con 0x7f17d80b4710 2024-03-29T11:28:53.647 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:53.641+0000 7f17bffff640 10 monclient(hunting): _negotiate 2024-03-29T11:28:53.647 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:53.641+0000 7f17bffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:28:53.647 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:53.641+0000 7f17bffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:28:53.647 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:53.641+0000 7f17bffff640 10 monclient(hunting): my global_id is 4616 2024-03-29T11:28:53.647 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:53.641+0000 7f17bffff640 1 -- 172.21.15.84:0/4069665930 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f17c4004ee0 con 0x7f17d80b4710 2024-03-29T11:28:53.648 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:53.641+0000 7f17bffff640 1 -- 172.21.15.84:0/4069665930 <== mon.0 v1:172.21.15.84:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1805238908 0 0) 0x7f17d81376a0 con 0x7f17d813f390 2024-03-29T11:28:53.648 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:53.641+0000 7f17bffff640 1 -- 172.21.15.84:0/4069665930 >> v1:172.21.15.84:6790/0 conn(0x7f17d80b4710 legacy=0x7f17d8133ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:53.648 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:53.641+0000 7f17bffff640 1 -- 172.21.15.84:0/4069665930 >> v1:172.21.15.179:6789/0 conn(0x7f17d80b3970 legacy=0x7f17d8137e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:53.648 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:53.641+0000 7f17bffff640 10 monclient: _finish_hunting 0 2024-03-29T11:28:53.648 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:53.641+0000 7f17bffff640 1 monclient: found mon.a 2024-03-29T11:28:53.648 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:53.641+0000 7f17bffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:28:53.648 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:53.641+0000 7f17bffff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.84:6789/0 2024-03-29T11:28:53.648 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:53.641+0000 7f17bffff640 1 -- 172.21.15.84:0/4069665930 --> v1:172.21.15.84:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17d8142710 con 0x7f17d813f390 2024-03-29T11:28:53.648 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:53.641+0000 7f17bffff640 10 monclient: _finish_auth 0 2024-03-29T11:28:53.648 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:53.641+0000 7f17bffff640 10 monclient: _check_auth_tickets 2024-03-29T11:28:53.648 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:53.641+0000 7f17bffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:28:53.648 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:53.641+0000 7f17ddf3d640 5 monclient: authenticate success, global_id 4554 2024-03-29T11:28:53.648 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:53.641+0000 7f17ddf3d640 10 monclient: _renew_subs 2024-03-29T11:28:53.648 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:53.641+0000 7f17ddf3d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.84:6789/0 2024-03-29T11:28:53.649 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:53.641+0000 7f17ddf3d640 1 -- 172.21.15.84:0/4069665930 --> v1:172.21.15.84:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f17d81e3110 con 0x7f17d813f390 2024-03-29T11:28:53.649 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:53.641+0000 7f17ddf3d640 10 client.4554.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:28:53.649 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:53.641+0000 7f17ddf3d640 10 monclient: _renew_subs 2024-03-29T11:28:53.649 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:53.641+0000 7f17ddf3d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.84:6789/0 2024-03-29T11:28:53.649 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:53.641+0000 7f17ddf3d640 1 -- 172.21.15.84:0/4069665930 --> v1:172.21.15.84:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f17d81e3480 con 0x7f17d813f390 2024-03-29T11:28:53.649 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:53.641+0000 7f17ddf3d640 1 librados: init done 2024-03-29T11:28:53.649 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:53.641+0000 7f17bffff640 1 -- 172.21.15.84:0/4069665930 <== mon.0 v1:172.21.15.84:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f17d00020f0 con 0x7f17d813f390 2024-03-29T11:28:53.649 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:53.641+0000 7f17bffff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:28:53.649 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:53.641+0000 7f17bffff640 1 -- 172.21.15.84:0/4069665930 <== mon.0 v1:172.21.15.84:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4236868694 0 0) 0x7f17d0003dc0 con 0x7f17d813f390 2024-03-29T11:28:53.649 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:53.641+0000 7f17bffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:28:53.649 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:53.641+0000 7f17bffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:28:53.649 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:53.641+0000 7f17bffff640 10 monclient: dump: 2024-03-29T11:28:53.649 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T11:28:53.649 INFO:teuthology.misc.health.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:28:53.649 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T11:18:53.864025+0000 2024-03-29T11:28:53.650 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T11:18:53.864025+0000 2024-03-29T11:28:53.650 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T11:28:53.650 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:28:53.650 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T11:28:53.650 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T11:28:53.650 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T11:28:53.650 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:28:53.650 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:53.641+0000 7f17d5ffb640 4 set_mon_vals no callback set 2024-03-29T11:28:53.650 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:53.643+0000 7f17ddf3d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:28:53.650 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:53.643+0000 7f17ddf3d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:28:53.650 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:53.643+0000 7f17ddf3d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.84:6789/0 2024-03-29T11:28:53.650 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:53.643+0000 7f17ddf3d640 1 -- 172.21.15.84:0/4069665930 --> v1:172.21.15.84:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f17d80688a0 con 0x7f17d813f390 2024-03-29T11:28:53.650 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:53.644+0000 7f17bffff640 1 -- 172.21.15.84:0/4069665930 <== mon.0 v1:172.21.15.84:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4033942909 0 0) 0x7f17d0019d90 con 0x7f17d813f390 2024-03-29T11:28:53.650 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:53.644+0000 7f17bffff640 1 -- 172.21.15.84:0/4069665930 <== mon.0 v1:172.21.15.84:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2808139587 0 0) 0x7f17d0083b20 con 0x7f17d813f390 2024-03-29T11:28:53.651 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:53.644+0000 7f17bffff640 10 client.4554.objecter ms_dispatch 0x7f17d8000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:28:53.651 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:53.644+0000 7f17bffff640 3 client.4554.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:28:53.651 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:53.644+0000 7f17bffff640 3 client.4554.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:28:53.651 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:53.645+0000 7f17bffff640 20 client.4554.objecter dump_active .. 0 homeless 2024-03-29T11:28:53.651 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:53.645+0000 7f17d77fe640 10 monclient: get_auth_request con 0x7f17c406e040 auth_method 0 2024-03-29T11:28:53.651 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:53.645+0000 7f17bffff640 10 client.4554.objecter ms_handle_connect 0x7f17c406e040 2024-03-29T11:28:53.651 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:53.652+0000 7f17bffff640 1 -- 172.21.15.84:0/4069665930 <== mon.0 v1:172.21.15.84:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f17d004f400 con 0x7f17d813f390 2024-03-29T11:28:53.651 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:53.652+0000 7f17bffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:28:53.652 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:53.652+0000 7f17bffff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:28:54.003 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:54.012+0000 7f17ddf3d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:28:54.003 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:54.012+0000 7f17ddf3d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:28:54.004 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:54.013+0000 7f17ddf3d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.84:6789/0 2024-03-29T11:28:54.004 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:54.013+0000 7f17ddf3d640 1 -- 172.21.15.84:0/4069665930 --> v1:172.21.15.84:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f17d81387b0 con 0x7f17d813f390 2024-03-29T11:28:54.004 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:54.013+0000 7f17bffff640 1 -- 172.21.15.84:0/4069665930 <== mon.0 v1:172.21.15.84:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4209477549) 0x7f17d81387b0 con 0x7f17d813f390 2024-03-29T11:28:54.004 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:54.013+0000 7f17bffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:28:54.004 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:54.013+0000 7f17bffff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:28:54.005 INFO:teuthology.misc.health.smithi084.stdout: 2024-03-29T11:28:54.005 INFO:teuthology.misc.health.smithi084.stdout:{"epoch":10,"fsid":"2c3e002f-9eb0-4860-853b-4ef1ef562213","created":"2024-03-29T11:19:32.029169+0000","modified":"2024-03-29T11:19:34.200299+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:34.200299+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"5c39e276-2034-4716-bbc0-57a634eefa92","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"cd580e99-109f-4bf1-88c0-a5987bc8b9ed","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"11e60a5a-051d-453f-a190-66387cb35592","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"2c329fbe-bbc4-4030-bf87-d711cad04679","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f4213d4a-bd36-4433-91ae-d7e2b02481d3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"f0cf44ea-6055-48af-805b-aed858102ff9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"d895eaa0-cc0b-4ee4-9b5a-2dfcd89a60b7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"9bc5a7bc-8f52-4db9-96d6-de5e68d52eaa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:28:54.008 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:54.017+0000 7f17ddf3d640 10 librados: watch_flush enter 2024-03-29T11:28:54.009 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:54.017+0000 7f17ddf3d640 10 librados: watch_flush exit 2024-03-29T11:28:54.009 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:54.017+0000 7f17ddf3d640 20 client.4554.objecter shutdown clearing up homeless session... 2024-03-29T11:28:54.009 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:54.017+0000 7f17ddf3d640 10 client.4554.objecter successfully canceled tick 2024-03-29T11:28:54.009 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:54.017+0000 7f17ddf3d640 1 -- 172.21.15.84:0/4069665930 >> v1:172.21.15.179:6816/2981134980 conn(0x7f17c406e040 legacy=0x7f17c4070500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:54.009 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:54.017+0000 7f17ddf3d640 10 monclient: shutdown 2024-03-29T11:28:54.010 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:54.017+0000 7f17ddf3d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:28:54.010 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:54.017+0000 7f17ddf3d640 1 -- 172.21.15.84:0/4069665930 >> v1:172.21.15.84:6789/0 conn(0x7f17d813f390 legacy=0x7f17d81341d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:54.010 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:54.017+0000 7f17ddf3d640 1 -- 172.21.15.84:0/4069665930 shutdown_connections 2024-03-29T11:28:54.010 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:54.017+0000 7f17ddf3d640 1 -- 172.21.15.84:0/4069665930 >> 172.21.15.84:0/4069665930 conn(0x7f17d8066a00 msgr2=0x7f17d80b1770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:28:54.010 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:54.017+0000 7f17ddf3d640 1 -- 172.21.15.84:0/4069665930 shutdown_connections 2024-03-29T11:28:54.010 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:54.017+0000 7f17ddf3d640 1 -- 172.21.15.84:0/4069665930 wait complete. 2024-03-29T11:28:54.010 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:28:54.018+0000 7f17ddf3d640 1 librados: shutdown 2024-03-29T11:28:54.023 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:29:00.830 DEBUG:teuthology.orchestra.run.smithi084:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:29:01.003 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:01.011+0000 7eff32365640 10 monclient: get_monmap_and_config 2024-03-29T11:29:01.003 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:01.011+0000 7eff32365640 10 monclient: build_initial_monmap 2024-03-29T11:29:01.003 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:01.011+0000 7eff32365640 10 monclient: monmap: 2024-03-29T11:29:01.003 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-29T11:29:01.003 INFO:teuthology.misc.health.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:29:01.004 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T11:29:01.012855+0000 2024-03-29T11:29:01.004 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T11:29:01.012855+0000 2024-03-29T11:29:01.004 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-29T11:29:01.004 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:29:01.004 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-29T11:29:01.004 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.noname-c 2024-03-29T11:29:01.004 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-b 2024-03-29T11:29:01.004 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:29:01.004 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:01.014+0000 7eff32365640 1 Processor -- start 2024-03-29T11:29:01.008 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:01.014+0000 7eff32365640 1 -- start start 2024-03-29T11:29:01.008 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:01.014+0000 7eff32365640 10 monclient: init 2024-03-29T11:29:01.008 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:01.014+0000 7eff32365640 10 monclient: _reopen_session rank -1 2024-03-29T11:29:01.008 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:01.014+0000 7eff32365640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:29:01.008 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:01.014+0000 7eff32365640 10 monclient(hunting): picked mon.noname-a con 0x7eff2c0b0770 addr v1:172.21.15.84:6789/0 2024-03-29T11:29:01.008 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:01.014+0000 7eff32365640 10 monclient(hunting): picked mon.noname-b con 0x7eff2c14a700 addr v1:172.21.15.84:6790/0 2024-03-29T11:29:01.008 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:01.014+0000 7eff32365640 10 monclient(hunting): picked mon.noname-c con 0x7eff2c0ac5a0 addr v1:172.21.15.179:6789/0 2024-03-29T11:29:01.008 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:01.014+0000 7eff32365640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7eff2c0aba70 con 0x7eff2c0b0770 2024-03-29T11:29:01.008 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:01.014+0000 7eff32365640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7eff2c0aecb0 con 0x7eff2c0ac5a0 2024-03-29T11:29:01.008 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:01.014+0000 7eff32365640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7eff2c0af150 con 0x7eff2c14a700 2024-03-29T11:29:01.008 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:01.014+0000 7eff32365640 10 monclient(hunting): _renew_subs 2024-03-29T11:29:01.008 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:01.014+0000 7eff32365640 10 monclient(hunting): authenticate will time out at 1402.309089s 2024-03-29T11:29:01.009 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:01.018+0000 7eff2b7fe640 1 --1- >> v1:172.21.15.84:6790/0 conn(0x7eff2c14a700 0x7eff2c0abe90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.84:6790/0 says I am v1:172.21.15.84:53382/0 (socket says 172.21.15.84:53382) 2024-03-29T11:29:01.009 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:01.018+0000 7eff2b7fe640 1 -- 172.21.15.84:0/3057525540 learned_addr learned my addr 172.21.15.84:0/3057525540 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-29T11:29:01.012 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:01.021+0000 7eff2affd640 1 -- 172.21.15.84:0/3057525540 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1577778544 0 0) 0x7eff2c0af150 con 0x7eff2c14a700 2024-03-29T11:29:01.012 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:01.021+0000 7eff2affd640 10 monclient(hunting): _negotiate 2024-03-29T11:29:01.012 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:01.021+0000 7eff2affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:29:01.012 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:01.021+0000 7eff2affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:29:01.012 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:01.021+0000 7eff2affd640 10 monclient(hunting): my global_id is 4619 2024-03-29T11:29:01.012 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:01.021+0000 7eff2affd640 1 -- 172.21.15.84:0/3057525540 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7eff14003520 con 0x7eff2c14a700 2024-03-29T11:29:01.012 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:01.021+0000 7eff2affd640 1 -- 172.21.15.84:0/3057525540 <== mon.1 v1:172.21.15.179:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3631756169 0 0) 0x7eff2c0aecb0 con 0x7eff2c0ac5a0 2024-03-29T11:29:01.012 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:01.021+0000 7eff2affd640 10 monclient(hunting): _negotiate 2024-03-29T11:29:01.012 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:01.021+0000 7eff2affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:29:01.013 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:01.021+0000 7eff2affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:29:01.013 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:01.021+0000 7eff2affd640 10 monclient(hunting): my global_id is 4504 2024-03-29T11:29:01.013 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:01.021+0000 7eff2affd640 1 -- 172.21.15.84:0/3057525540 --> v1:172.21.15.179:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7eff2c0af150 con 0x7eff2c0ac5a0 2024-03-29T11:29:01.013 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:01.021+0000 7eff2affd640 1 -- 172.21.15.84:0/3057525540 <== mon.0 v1:172.21.15.84:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 637911172 0 0) 0x7eff2c0aba70 con 0x7eff2c0b0770 2024-03-29T11:29:01.013 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:01.021+0000 7eff2affd640 10 monclient(hunting): _negotiate 2024-03-29T11:29:01.013 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:01.021+0000 7eff2affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:29:01.013 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:01.021+0000 7eff2affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:29:01.013 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:01.021+0000 7eff2affd640 10 monclient(hunting): my global_id is 4557 2024-03-29T11:29:01.013 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:01.021+0000 7eff2affd640 1 -- 172.21.15.84:0/3057525540 --> v1:172.21.15.84:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7eff2c0aecb0 con 0x7eff2c0b0770 2024-03-29T11:29:01.014 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:01.022+0000 7eff2affd640 1 -- 172.21.15.84:0/3057525540 <== mon.0 v1:172.21.15.84:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3441166865 0 0) 0x7eff2c0aecb0 con 0x7eff2c0b0770 2024-03-29T11:29:01.014 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:01.022+0000 7eff2affd640 1 -- 172.21.15.84:0/3057525540 >> v1:172.21.15.84:6790/0 conn(0x7eff2c14a700 legacy=0x7eff2c0abe90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:01.014 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:01.022+0000 7eff2affd640 1 -- 172.21.15.84:0/3057525540 >> v1:172.21.15.179:6789/0 conn(0x7eff2c0ac5a0 legacy=0x7eff2c151f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:01.015 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:01.022+0000 7eff2affd640 10 monclient: _finish_hunting 0 2024-03-29T11:29:01.015 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:01.022+0000 7eff2affd640 1 monclient: found mon.noname-a 2024-03-29T11:29:01.015 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:01.022+0000 7eff2affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:29:01.015 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:01.022+0000 7eff2affd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.84:6789/0 2024-03-29T11:29:01.015 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:01.022+0000 7eff2affd640 1 -- 172.21.15.84:0/3057525540 --> v1:172.21.15.84:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff2c0ac9d0 con 0x7eff2c0b0770 2024-03-29T11:29:01.015 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:01.022+0000 7eff2affd640 10 monclient: _finish_auth 0 2024-03-29T11:29:01.015 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:01.022+0000 7eff2affd640 10 monclient: _check_auth_tickets 2024-03-29T11:29:01.015 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:01.022+0000 7eff2affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:29:01.015 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:01.022+0000 7eff2affd640 1 -- 172.21.15.84:0/3057525540 <== mon.0 v1:172.21.15.84:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4236868694 0 0) 0x7eff1c001530 con 0x7eff2c0b0770 2024-03-29T11:29:01.015 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:01.022+0000 7eff2affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:29:01.015 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:01.022+0000 7eff2affd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:29:01.015 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:01.022+0000 7eff2affd640 10 monclient: dump: 2024-03-29T11:29:01.015 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T11:29:01.015 INFO:teuthology.misc.health.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:29:01.015 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T11:18:53.864025+0000 2024-03-29T11:29:01.016 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T11:18:53.864025+0000 2024-03-29T11:29:01.016 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T11:29:01.016 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:29:01.016 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T11:29:01.016 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T11:29:01.016 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T11:29:01.016 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:29:01.016 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:01.022+0000 7eff2affd640 1 -- 172.21.15.84:0/3057525540 <== mon.0 v1:172.21.15.84:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7eff1c001780 con 0x7eff2c0b0770 2024-03-29T11:29:01.016 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:01.022+0000 7eff2affd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:29:01.016 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:01.022+0000 7eff2affd640 1 -- 172.21.15.84:0/3057525540 <== mon.0 v1:172.21.15.84:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4236868694 0 0) 0x7eff1c001b10 con 0x7eff2c0b0770 2024-03-29T11:29:01.016 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:01.022+0000 7eff2affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:29:01.016 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:01.022+0000 7eff2affd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:29:01.016 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:01.022+0000 7eff2affd640 10 monclient: dump: 2024-03-29T11:29:01.016 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T11:29:01.016 INFO:teuthology.misc.health.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:29:01.017 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T11:18:53.864025+0000 2024-03-29T11:29:01.017 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T11:18:53.864025+0000 2024-03-29T11:29:01.017 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T11:29:01.017 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:29:01.017 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T11:29:01.017 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T11:29:01.017 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T11:29:01.017 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:29:01.017 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:01.023+0000 7eff32365640 5 monclient: authenticate success, global_id 4557 2024-03-29T11:29:01.017 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:01.023+0000 7eff32365640 10 monclient: get_monmap_and_config success 2024-03-29T11:29:01.017 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:01.023+0000 7eff32365640 4 set_mon_vals no callback set 2024-03-29T11:29:01.017 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:01.024+0000 7eff32365640 10 monclient: shutdown 2024-03-29T11:29:01.017 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:01.024+0000 7eff32365640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:29:01.017 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:01.024+0000 7eff32365640 1 -- 172.21.15.84:0/3057525540 >> v1:172.21.15.84:6789/0 conn(0x7eff2c0b0770 legacy=0x7eff2c149ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:01.018 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:01.025+0000 7eff32365640 1 -- 172.21.15.84:0/3057525540 shutdown_connections 2024-03-29T11:29:01.018 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:01.025+0000 7eff32365640 1 -- 172.21.15.84:0/3057525540 >> 172.21.15.84:0/3057525540 conn(0x7eff2c06b660 msgr2=0x7eff2c06ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:29:01.018 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:01.025+0000 7eff32365640 1 -- 172.21.15.84:0/3057525540 shutdown_connections 2024-03-29T11:29:01.018 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:01.025+0000 7eff32365640 1 -- 172.21.15.84:0/3057525540 wait complete. 2024-03-29T11:29:01.018 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:01.026+0000 7eff32365640 10 monclient: build_initial_monmap 2024-03-29T11:29:01.019 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:01.026+0000 7eff32365640 10 monclient: monmap: 2024-03-29T11:29:01.019 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-29T11:29:01.019 INFO:teuthology.misc.health.smithi084.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:29:01.019 INFO:teuthology.misc.health.smithi084.stderr:last_changed 0.000000 2024-03-29T11:29:01.019 INFO:teuthology.misc.health.smithi084.stderr:created 0.000000 2024-03-29T11:29:01.019 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-29T11:29:01.019 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:29:01.019 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-29T11:29:01.019 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.noname-b 2024-03-29T11:29:01.019 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-c 2024-03-29T11:29:01.019 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:29:01.019 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:01.026+0000 7eff32365640 1 librados: starting msgr at 2024-03-29T11:29:01.019 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:01.026+0000 7eff32365640 1 librados: starting objecter 2024-03-29T11:29:01.019 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:01.027+0000 7eff32365640 1 Processor -- start 2024-03-29T11:29:01.020 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:01.027+0000 7eff32365640 1 -- start start 2024-03-29T11:29:01.020 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:01.027+0000 7eff32365640 1 librados: setting wanted keys 2024-03-29T11:29:01.020 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:01.027+0000 7eff32365640 1 librados: calling monclient init 2024-03-29T11:29:01.020 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:01.027+0000 7eff32365640 10 monclient: init 2024-03-29T11:29:01.021 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:01.029+0000 7eff32365640 10 monclient: _reopen_session rank -1 2024-03-29T11:29:01.021 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:01.029+0000 7eff32365640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:29:01.022 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:01.029+0000 7eff32365640 10 monclient(hunting): picked mon.noname-a con 0x7eff2c0ac5a0 addr v1:172.21.15.84:6789/0 2024-03-29T11:29:01.022 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:01.029+0000 7eff32365640 10 monclient(hunting): picked mon.noname-b con 0x7eff2c0b0770 addr v1:172.21.15.179:6789/0 2024-03-29T11:29:01.022 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:01.029+0000 7eff32365640 10 monclient(hunting): picked mon.noname-c con 0x7eff2c14a700 addr v1:172.21.15.84:6790/0 2024-03-29T11:29:01.022 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:01.029+0000 7eff32365640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7eff2c07aff0 con 0x7eff2c0ac5a0 2024-03-29T11:29:01.022 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:01.029+0000 7eff32365640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7eff2c07d8e0 con 0x7eff2c0b0770 2024-03-29T11:29:01.022 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:01.029+0000 7eff32365640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7eff2c08be40 con 0x7eff2c14a700 2024-03-29T11:29:01.022 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:01.029+0000 7eff32365640 10 monclient(hunting): _renew_subs 2024-03-29T11:29:01.022 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:01.029+0000 7eff32365640 10 monclient(hunting): authenticate will time out at 1402.323447s 2024-03-29T11:29:01.022 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:01.029+0000 7eff30902640 1 --1- >> v1:172.21.15.84:6790/0 conn(0x7eff2c14a700 0x7eff2c1f82d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.84:6790/0 says I am v1:172.21.15.84:56696/0 (socket says 172.21.15.84:56696) 2024-03-29T11:29:01.022 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:01.030+0000 7eff30902640 1 -- 172.21.15.84:0/3574806910 learned_addr learned my addr 172.21.15.84:0/3574806910 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-29T11:29:01.022 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:01.030+0000 7eff28ff9640 10 client.?.objecter ms_handle_connect 0x7eff2c14a700 2024-03-29T11:29:01.022 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:01.030+0000 7eff28ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:29:01.023 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:01.033+0000 7eff28ff9640 1 -- 172.21.15.84:0/3574806910 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3942435627 0 0) 0x7eff2c08be40 con 0x7eff2c14a700 2024-03-29T11:29:01.023 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:01.033+0000 7eff28ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:29:01.023 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:01.033+0000 7eff28ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:29:01.023 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:01.033+0000 7eff28ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:29:01.023 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:01.033+0000 7eff28ff9640 10 monclient(hunting): my global_id is 4622 2024-03-29T11:29:01.023 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:01.033+0000 7eff28ff9640 1 -- 172.21.15.84:0/3574806910 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7eff18004eb0 con 0x7eff2c14a700 2024-03-29T11:29:01.028 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:01.035+0000 7eff28ff9640 10 client.?.objecter ms_handle_connect 0x7eff2c0ac5a0 2024-03-29T11:29:01.028 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:01.035+0000 7eff28ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:29:01.028 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:01.035+0000 7eff28ff9640 10 client.?.objecter ms_handle_connect 0x7eff2c0b0770 2024-03-29T11:29:01.028 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:01.035+0000 7eff28ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:29:01.028 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:01.035+0000 7eff28ff9640 1 -- 172.21.15.84:0/3574806910 <== mon.2 v1:172.21.15.84:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4015151925 0 0) 0x7eff18004eb0 con 0x7eff2c14a700 2024-03-29T11:29:01.028 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:01.035+0000 7eff28ff9640 1 -- 172.21.15.84:0/3574806910 --> v1:172.21.15.84:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7eff2c08be40 con 0x7eff2c14a700 2024-03-29T11:29:01.028 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:01.035+0000 7eff28ff9640 1 -- 172.21.15.84:0/3574806910 <== mon.2 v1:172.21.15.84:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4236868694 0 0) 0x7eff200018f0 con 0x7eff2c14a700 2024-03-29T11:29:01.028 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:01.035+0000 7eff28ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:29:01.029 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:01.035+0000 7eff28ff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:29:01.029 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:01.035+0000 7eff28ff9640 10 monclient(hunting): dump: 2024-03-29T11:29:01.029 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T11:29:01.029 INFO:teuthology.misc.health.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:29:01.029 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T11:18:53.864025+0000 2024-03-29T11:29:01.029 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T11:18:53.864025+0000 2024-03-29T11:29:01.029 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T11:29:01.029 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:29:01.029 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T11:29:01.029 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T11:29:01.029 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T11:29:01.029 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:29:01.029 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:01.035+0000 7eff28ff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:29:01.029 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:01.036+0000 7eff28ff9640 1 -- 172.21.15.84:0/3574806910 <== mon.0 v1:172.21.15.84:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4116161332 0 0) 0x7eff2c07aff0 con 0x7eff2c0ac5a0 2024-03-29T11:29:01.029 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:01.036+0000 7eff28ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:29:01.030 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:01.036+0000 7eff28ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:29:01.030 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:01.036+0000 7eff28ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:29:01.030 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:01.036+0000 7eff28ff9640 10 monclient(hunting): my global_id is 4560 2024-03-29T11:29:01.030 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:01.036+0000 7eff28ff9640 1 -- 172.21.15.84:0/3574806910 --> v1:172.21.15.84:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7eff18004eb0 con 0x7eff2c0ac5a0 2024-03-29T11:29:01.030 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:01.036+0000 7eff28ff9640 1 -- 172.21.15.84:0/3574806910 <== mon.1 v1:172.21.15.179:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3770291472 0 0) 0x7eff2c07d8e0 con 0x7eff2c0b0770 2024-03-29T11:29:01.030 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:01.036+0000 7eff28ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:29:01.030 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:01.036+0000 7eff28ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:29:01.030 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:01.036+0000 7eff28ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:29:01.030 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:01.036+0000 7eff28ff9640 10 monclient(hunting): my global_id is 4507 2024-03-29T11:29:01.030 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:01.036+0000 7eff28ff9640 1 -- 172.21.15.84:0/3574806910 --> v1:172.21.15.179:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7eff2c07aff0 con 0x7eff2c0b0770 2024-03-29T11:29:01.033 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:01.037+0000 7eff28ff9640 1 -- 172.21.15.84:0/3574806910 <== mon.2 v1:172.21.15.84:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 493962722 0 0) 0x7eff2c08be40 con 0x7eff2c14a700 2024-03-29T11:29:01.033 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:01.037+0000 7eff28ff9640 1 -- 172.21.15.84:0/3574806910 >> v1:172.21.15.179:6789/0 conn(0x7eff2c0b0770 legacy=0x7eff2c07dd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:01.033 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:01.037+0000 7eff28ff9640 1 -- 172.21.15.84:0/3574806910 >> v1:172.21.15.84:6789/0 conn(0x7eff2c0ac5a0 legacy=0x7eff2c1e9a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:01.033 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:01.037+0000 7eff28ff9640 10 monclient: _finish_hunting 0 2024-03-29T11:29:01.033 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:01.037+0000 7eff28ff9640 1 monclient: found mon.c 2024-03-29T11:29:01.033 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:01.037+0000 7eff28ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:29:01.033 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:01.037+0000 7eff28ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-29T11:29:01.033 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:01.037+0000 7eff28ff9640 1 -- 172.21.15.84:0/3574806910 --> v1:172.21.15.84:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff2c07f830 con 0x7eff2c14a700 2024-03-29T11:29:01.033 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:01.037+0000 7eff28ff9640 10 monclient: _finish_auth 0 2024-03-29T11:29:01.033 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:01.037+0000 7eff28ff9640 10 monclient: _check_auth_tickets 2024-03-29T11:29:01.034 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:01.037+0000 7eff28ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:29:01.034 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:01.037+0000 7eff32365640 5 monclient: authenticate success, global_id 4622 2024-03-29T11:29:01.034 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:01.037+0000 7eff32365640 10 monclient: _renew_subs 2024-03-29T11:29:01.034 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:01.037+0000 7eff32365640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-29T11:29:01.034 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:01.037+0000 7eff32365640 1 -- 172.21.15.84:0/3574806910 --> v1:172.21.15.84:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7eff2c07fb10 con 0x7eff2c14a700 2024-03-29T11:29:01.034 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:01.037+0000 7eff32365640 10 client.4622.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:29:01.034 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:01.037+0000 7eff32365640 10 monclient: _renew_subs 2024-03-29T11:29:01.034 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:01.037+0000 7eff32365640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-29T11:29:01.034 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:01.037+0000 7eff32365640 1 -- 172.21.15.84:0/3574806910 --> v1:172.21.15.84:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7eff2c1ea110 con 0x7eff2c14a700 2024-03-29T11:29:01.034 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:01.037+0000 7eff32365640 1 librados: init done 2024-03-29T11:29:01.034 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:01.040+0000 7eff32365640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:29:01.034 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:01.040+0000 7eff32365640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:29:01.034 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:01.040+0000 7eff32365640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-29T11:29:01.034 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:01.040+0000 7eff32365640 1 -- 172.21.15.84:0/3574806910 --> v1:172.21.15.84:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7eff2c0ad7f0 con 0x7eff2c14a700 2024-03-29T11:29:01.034 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:01.042+0000 7eff28ff9640 1 -- 172.21.15.84:0/3574806910 <== mon.2 v1:172.21.15.84:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7eff20003610 con 0x7eff2c14a700 2024-03-29T11:29:01.035 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:01.042+0000 7eff28ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:29:01.035 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:01.042+0000 7eff28ff9640 1 -- 172.21.15.84:0/3574806910 <== mon.2 v1:172.21.15.84:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4236868694 0 0) 0x7eff20003a20 con 0x7eff2c14a700 2024-03-29T11:29:01.035 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:01.042+0000 7eff28ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:29:01.035 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:01.042+0000 7eff28ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:29:01.035 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:01.042+0000 7eff28ff9640 10 monclient: dump: 2024-03-29T11:29:01.035 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T11:29:01.035 INFO:teuthology.misc.health.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:29:01.035 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T11:18:53.864025+0000 2024-03-29T11:29:01.035 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T11:18:53.864025+0000 2024-03-29T11:29:01.035 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T11:29:01.035 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:29:01.035 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T11:29:01.035 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T11:29:01.035 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T11:29:01.035 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:29:01.036 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:01.042+0000 7eff29ffb640 4 set_mon_vals no callback set 2024-03-29T11:29:01.036 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:01.045+0000 7eff28ff9640 1 -- 172.21.15.84:0/3574806910 <== mon.2 v1:172.21.15.84:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4033942909 0 0) 0x7eff20003b80 con 0x7eff2c14a700 2024-03-29T11:29:01.038 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:01.046+0000 7eff28ff9640 1 -- 172.21.15.84:0/3574806910 <== mon.2 v1:172.21.15.84:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2808139587 0 0) 0x7eff20084a10 con 0x7eff2c14a700 2024-03-29T11:29:01.038 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:01.046+0000 7eff28ff9640 10 client.4622.objecter ms_dispatch 0x7eff2c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:29:01.038 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:01.046+0000 7eff28ff9640 3 client.4622.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:29:01.038 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:01.046+0000 7eff28ff9640 3 client.4622.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:29:01.038 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:01.046+0000 7eff28ff9640 20 client.4622.objecter dump_active .. 0 homeless 2024-03-29T11:29:01.039 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:01.047+0000 7eff2bfff640 10 monclient: get_auth_request con 0x7eff1806ded0 auth_method 0 2024-03-29T11:29:01.039 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:01.048+0000 7eff28ff9640 10 client.4622.objecter ms_handle_connect 0x7eff1806ded0 2024-03-29T11:29:01.043 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:01.052+0000 7eff28ff9640 1 -- 172.21.15.84:0/3574806910 <== mon.2 v1:172.21.15.84:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7eff200502f0 con 0x7eff2c14a700 2024-03-29T11:29:01.043 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:01.052+0000 7eff28ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:29:01.043 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:01.052+0000 7eff28ff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:29:01.416 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:01.426+0000 7eff32365640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:29:01.417 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:01.426+0000 7eff32365640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:29:01.417 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:01.426+0000 7eff32365640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-29T11:29:01.417 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:01.426+0000 7eff32365640 1 -- 172.21.15.84:0/3574806910 --> v1:172.21.15.84:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7eff2c075f10 con 0x7eff2c14a700 2024-03-29T11:29:01.417 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:01.427+0000 7eff28ff9640 1 -- 172.21.15.84:0/3574806910 <== mon.2 v1:172.21.15.84:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4209477549) 0x7eff20052220 con 0x7eff2c14a700 2024-03-29T11:29:01.417 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:01.427+0000 7eff28ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:29:01.417 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:01.427+0000 7eff28ff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:29:01.418 INFO:teuthology.misc.health.smithi084.stdout: 2024-03-29T11:29:01.419 INFO:teuthology.misc.health.smithi084.stdout:{"epoch":10,"fsid":"2c3e002f-9eb0-4860-853b-4ef1ef562213","created":"2024-03-29T11:19:32.029169+0000","modified":"2024-03-29T11:19:34.200299+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:34.200299+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"5c39e276-2034-4716-bbc0-57a634eefa92","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"cd580e99-109f-4bf1-88c0-a5987bc8b9ed","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"11e60a5a-051d-453f-a190-66387cb35592","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"2c329fbe-bbc4-4030-bf87-d711cad04679","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f4213d4a-bd36-4433-91ae-d7e2b02481d3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"f0cf44ea-6055-48af-805b-aed858102ff9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"d895eaa0-cc0b-4ee4-9b5a-2dfcd89a60b7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"9bc5a7bc-8f52-4db9-96d6-de5e68d52eaa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:29:01.422 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:01.431+0000 7eff32365640 10 librados: watch_flush enter 2024-03-29T11:29:01.422 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:01.431+0000 7eff32365640 10 librados: watch_flush exit 2024-03-29T11:29:01.422 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:01.431+0000 7eff32365640 20 client.4622.objecter shutdown clearing up homeless session... 2024-03-29T11:29:01.422 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:01.431+0000 7eff32365640 10 client.4622.objecter successfully canceled tick 2024-03-29T11:29:01.423 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:01.431+0000 7eff32365640 1 -- 172.21.15.84:0/3574806910 >> v1:172.21.15.179:6816/2981134980 conn(0x7eff1806ded0 legacy=0x7eff18070390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:01.423 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:01.431+0000 7eff32365640 10 monclient: shutdown 2024-03-29T11:29:01.423 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:01.432+0000 7eff32365640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:29:01.423 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:01.432+0000 7eff32365640 1 -- 172.21.15.84:0/3574806910 >> v1:172.21.15.84:6790/0 conn(0x7eff2c14a700 legacy=0x7eff2c1f82d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:01.423 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:01.432+0000 7eff32365640 1 -- 172.21.15.84:0/3574806910 shutdown_connections 2024-03-29T11:29:01.423 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:01.432+0000 7eff32365640 1 -- 172.21.15.84:0/3574806910 >> 172.21.15.84:0/3574806910 conn(0x7eff2c06b660 msgr2=0x7eff2c06ba40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:29:01.423 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:01.432+0000 7eff32365640 1 -- 172.21.15.84:0/3574806910 shutdown_connections 2024-03-29T11:29:01.423 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:01.432+0000 7eff32365640 1 -- 172.21.15.84:0/3574806910 wait complete. 2024-03-29T11:29:01.423 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:01.432+0000 7eff32365640 1 librados: shutdown 2024-03-29T11:29:01.442 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:29:08.250 DEBUG:teuthology.orchestra.run.smithi084:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:29:08.410 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:08.419+0000 7f5f53915640 10 monclient: get_monmap_and_config 2024-03-29T11:29:08.410 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:08.419+0000 7f5f53915640 10 monclient: build_initial_monmap 2024-03-29T11:29:08.410 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:08.419+0000 7f5f53915640 10 monclient: monmap: 2024-03-29T11:29:08.410 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-29T11:29:08.410 INFO:teuthology.misc.health.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:29:08.410 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T11:29:08.420099+0000 2024-03-29T11:29:08.410 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T11:29:08.420099+0000 2024-03-29T11:29:08.410 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-29T11:29:08.410 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:29:08.410 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-29T11:29:08.410 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.noname-c 2024-03-29T11:29:08.411 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-b 2024-03-29T11:29:08.411 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:29:08.412 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:08.420+0000 7f5f53915640 1 Processor -- start 2024-03-29T11:29:08.412 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:08.420+0000 7f5f53915640 1 -- start start 2024-03-29T11:29:08.412 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:08.420+0000 7f5f53915640 10 monclient: init 2024-03-29T11:29:08.412 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:08.421+0000 7f5f53915640 10 monclient: _reopen_session rank -1 2024-03-29T11:29:08.412 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:08.421+0000 7f5f53915640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:29:08.412 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:08.421+0000 7f5f53915640 10 monclient(hunting): picked mon.noname-b con 0x7f5f4c0b0770 addr v1:172.21.15.84:6790/0 2024-03-29T11:29:08.415 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:08.421+0000 7f5f53915640 10 monclient(hunting): picked mon.noname-c con 0x7f5f4c14a700 addr v1:172.21.15.179:6789/0 2024-03-29T11:29:08.415 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:08.421+0000 7f5f53915640 10 monclient(hunting): picked mon.noname-a con 0x7f5f4c0ac5a0 addr v1:172.21.15.84:6789/0 2024-03-29T11:29:08.416 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:08.421+0000 7f5f53915640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5f4c0aba70 con 0x7f5f4c0ac5a0 2024-03-29T11:29:08.416 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:08.421+0000 7f5f53915640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5f4c0aecb0 con 0x7f5f4c14a700 2024-03-29T11:29:08.416 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:08.421+0000 7f5f53915640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5f4c0af150 con 0x7f5f4c0b0770 2024-03-29T11:29:08.416 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:08.421+0000 7f5f53915640 10 monclient(hunting): _renew_subs 2024-03-29T11:29:08.416 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:08.421+0000 7f5f53915640 10 monclient(hunting): authenticate will time out at 1409.715736s 2024-03-29T11:29:08.416 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:08.425+0000 7f5f51eb2640 1 --1- >> v1:172.21.15.84:6789/0 conn(0x7f5f4c0ac5a0 0x7f5f4c151f00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.84:6789/0 says I am v1:172.21.15.84:33276/0 (socket says 172.21.15.84:33276) 2024-03-29T11:29:08.416 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:08.425+0000 7f5f51eb2640 1 -- 172.21.15.84:0/1359800350 learned_addr learned my addr 172.21.15.84:0/1359800350 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-29T11:29:08.417 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:08.426+0000 7f5f43fff640 1 -- 172.21.15.84:0/1359800350 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1165736439 0 0) 0x7f5f4c0af150 con 0x7f5f4c0b0770 2024-03-29T11:29:08.417 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:08.426+0000 7f5f43fff640 10 monclient(hunting): _negotiate 2024-03-29T11:29:08.417 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:08.426+0000 7f5f43fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:29:08.418 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:08.426+0000 7f5f43fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:29:08.418 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:08.426+0000 7f5f43fff640 10 monclient(hunting): my global_id is 4625 2024-03-29T11:29:08.418 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:08.426+0000 7f5f43fff640 1 -- 172.21.15.84:0/1359800350 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5f38003520 con 0x7f5f4c0b0770 2024-03-29T11:29:08.419 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:08.428+0000 7f5f43fff640 1 -- 172.21.15.84:0/1359800350 <== mon.0 v1:172.21.15.84:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1166386100 0 0) 0x7f5f4c0aba70 con 0x7f5f4c0ac5a0 2024-03-29T11:29:08.419 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:08.428+0000 7f5f43fff640 10 monclient(hunting): _negotiate 2024-03-29T11:29:08.419 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:08.428+0000 7f5f43fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:29:08.420 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:08.428+0000 7f5f43fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:29:08.420 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:08.428+0000 7f5f43fff640 10 monclient(hunting): my global_id is 4563 2024-03-29T11:29:08.420 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:08.428+0000 7f5f43fff640 1 -- 172.21.15.84:0/1359800350 --> v1:172.21.15.84:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5f38003840 con 0x7f5f4c0ac5a0 2024-03-29T11:29:08.420 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:08.428+0000 7f5f43fff640 1 -- 172.21.15.84:0/1359800350 <== mon.2 v1:172.21.15.84:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 766544136 0 0) 0x7f5f38003520 con 0x7f5f4c0b0770 2024-03-29T11:29:08.420 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:08.428+0000 7f5f43fff640 1 -- 172.21.15.84:0/1359800350 >> v1:172.21.15.179:6789/0 conn(0x7f5f4c14a700 legacy=0x7f5f4c0abe90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:08.420 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:08.428+0000 7f5f43fff640 1 -- 172.21.15.84:0/1359800350 >> v1:172.21.15.84:6789/0 conn(0x7f5f4c0ac5a0 legacy=0x7f5f4c151f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:08.420 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:08.428+0000 7f5f43fff640 10 monclient: _finish_hunting 0 2024-03-29T11:29:08.420 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:08.428+0000 7f5f43fff640 1 monclient: found mon.noname-b 2024-03-29T11:29:08.420 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:08.428+0000 7f5f43fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:29:08.420 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:08.428+0000 7f5f43fff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.84:6790/0 2024-03-29T11:29:08.420 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:08.428+0000 7f5f43fff640 1 -- 172.21.15.84:0/1359800350 --> v1:172.21.15.84:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f4c0ac9d0 con 0x7f5f4c0b0770 2024-03-29T11:29:08.420 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:08.428+0000 7f5f43fff640 10 monclient: _finish_auth 0 2024-03-29T11:29:08.420 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:08.428+0000 7f5f43fff640 10 monclient: _check_auth_tickets 2024-03-29T11:29:08.420 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:08.428+0000 7f5f43fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:29:08.420 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:08.428+0000 7f5f43fff640 1 -- 172.21.15.84:0/1359800350 <== mon.2 v1:172.21.15.84:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4236868694 0 0) 0x7f5f480015f0 con 0x7f5f4c0b0770 2024-03-29T11:29:08.421 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:08.428+0000 7f5f43fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:29:08.421 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:08.428+0000 7f5f43fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:29:08.421 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:08.428+0000 7f5f43fff640 10 monclient: dump: 2024-03-29T11:29:08.421 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T11:29:08.421 INFO:teuthology.misc.health.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:29:08.421 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T11:18:53.864025+0000 2024-03-29T11:29:08.421 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T11:18:53.864025+0000 2024-03-29T11:29:08.421 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T11:29:08.421 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:29:08.421 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T11:29:08.421 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T11:29:08.421 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T11:29:08.421 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:29:08.421 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:08.428+0000 7f5f53915640 5 monclient: authenticate success, global_id 4625 2024-03-29T11:29:08.421 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:08.428+0000 7f5f53915640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:29:08.422 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:08.428+0000 7f5f43fff640 1 -- 172.21.15.84:0/1359800350 <== mon.2 v1:172.21.15.84:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5f48001750 con 0x7f5f4c0b0770 2024-03-29T11:29:08.422 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:08.428+0000 7f5f43fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:29:08.422 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:08.428+0000 7f5f53915640 10 monclient: get_monmap_and_config success 2024-03-29T11:29:08.422 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:08.428+0000 7f5f53915640 4 set_mon_vals no callback set 2024-03-29T11:29:08.422 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:08.428+0000 7f5f43fff640 1 -- 172.21.15.84:0/1359800350 <== mon.2 v1:172.21.15.84:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4236868694 0 0) 0x7f5f48001ba0 con 0x7f5f4c0b0770 2024-03-29T11:29:08.422 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:08.428+0000 7f5f43fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:29:08.422 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:08.428+0000 7f5f43fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:29:08.422 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:08.428+0000 7f5f43fff640 10 monclient: dump: 2024-03-29T11:29:08.422 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T11:29:08.422 INFO:teuthology.misc.health.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:29:08.422 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T11:18:53.864025+0000 2024-03-29T11:29:08.422 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T11:18:53.864025+0000 2024-03-29T11:29:08.422 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T11:29:08.423 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:29:08.423 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T11:29:08.423 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T11:29:08.423 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T11:29:08.423 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:29:08.424 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:08.429+0000 7f5f53915640 10 monclient: shutdown 2024-03-29T11:29:08.424 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:08.429+0000 7f5f53915640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:29:08.424 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:08.429+0000 7f5f53915640 1 -- 172.21.15.84:0/1359800350 >> v1:172.21.15.84:6790/0 conn(0x7f5f4c0b0770 legacy=0x7f5f4c149ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:08.424 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:08.430+0000 7f5f53915640 1 -- 172.21.15.84:0/1359800350 shutdown_connections 2024-03-29T11:29:08.424 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:08.430+0000 7f5f53915640 1 -- 172.21.15.84:0/1359800350 >> 172.21.15.84:0/1359800350 conn(0x7f5f4c06b660 msgr2=0x7f5f4c06ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:29:08.424 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:08.433+0000 7f5f53915640 1 -- 172.21.15.84:0/1359800350 shutdown_connections 2024-03-29T11:29:08.424 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:08.433+0000 7f5f53915640 1 -- 172.21.15.84:0/1359800350 wait complete. 2024-03-29T11:29:08.426 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:08.434+0000 7f5f53915640 10 monclient: build_initial_monmap 2024-03-29T11:29:08.426 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:08.434+0000 7f5f53915640 10 monclient: monmap: 2024-03-29T11:29:08.426 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-29T11:29:08.426 INFO:teuthology.misc.health.smithi084.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:29:08.426 INFO:teuthology.misc.health.smithi084.stderr:last_changed 0.000000 2024-03-29T11:29:08.426 INFO:teuthology.misc.health.smithi084.stderr:created 0.000000 2024-03-29T11:29:08.427 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-29T11:29:08.427 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:29:08.427 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-29T11:29:08.427 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.noname-b 2024-03-29T11:29:08.427 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-c 2024-03-29T11:29:08.427 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:29:08.427 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:08.435+0000 7f5f53915640 1 librados: starting msgr at 2024-03-29T11:29:08.427 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:08.435+0000 7f5f53915640 1 librados: starting objecter 2024-03-29T11:29:08.427 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:08.435+0000 7f5f53915640 1 Processor -- start 2024-03-29T11:29:08.427 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:08.435+0000 7f5f53915640 1 -- start start 2024-03-29T11:29:08.427 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:08.435+0000 7f5f53915640 1 librados: setting wanted keys 2024-03-29T11:29:08.427 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:08.435+0000 7f5f53915640 1 librados: calling monclient init 2024-03-29T11:29:08.427 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:08.435+0000 7f5f53915640 10 monclient: init 2024-03-29T11:29:08.429 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:08.436+0000 7f5f53915640 10 monclient: _reopen_session rank -1 2024-03-29T11:29:08.429 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:08.436+0000 7f5f53915640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:29:08.429 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:08.436+0000 7f5f53915640 10 monclient(hunting): picked mon.noname-c con 0x7f5f4c0ac5a0 addr v1:172.21.15.84:6790/0 2024-03-29T11:29:08.429 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:08.436+0000 7f5f53915640 10 monclient(hunting): picked mon.noname-b con 0x7f5f4c0b0770 addr v1:172.21.15.179:6789/0 2024-03-29T11:29:08.429 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:08.436+0000 7f5f53915640 10 monclient(hunting): picked mon.noname-a con 0x7f5f4c14a700 addr v1:172.21.15.84:6789/0 2024-03-29T11:29:08.429 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:08.436+0000 7f5f53915640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5f4c144ae0 con 0x7f5f4c14a700 2024-03-29T11:29:08.429 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:08.436+0000 7f5f53915640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5f4c125d60 con 0x7f5f4c0b0770 2024-03-29T11:29:08.429 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:08.436+0000 7f5f53915640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5f4c126280 con 0x7f5f4c0ac5a0 2024-03-29T11:29:08.429 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:08.436+0000 7f5f53915640 10 monclient(hunting): _renew_subs 2024-03-29T11:29:08.429 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:08.436+0000 7f5f53915640 10 monclient(hunting): authenticate will time out at 1409.730533s 2024-03-29T11:29:08.432 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:08.439+0000 7f5f51eb2640 1 --1- >> v1:172.21.15.84:6789/0 conn(0x7f5f4c14a700 0x7f5f4c1e6f00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.84:6789/0 says I am v1:172.21.15.84:33982/0 (socket says 172.21.15.84:33982) 2024-03-29T11:29:08.432 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:08.439+0000 7f5f50eb0640 1 --1- >> v1:172.21.15.179:6789/0 conn(0x7f5f4c0b0770 0x7f5f4c143dd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.179:6789/0 says I am v1:172.21.15.84:48491/0 (socket says 172.21.15.84:48491) 2024-03-29T11:29:08.432 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:08.439+0000 7f5f50eb0640 1 -- 172.21.15.84:0/3875439917 learned_addr learned my addr 172.21.15.84:0/3875439917 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-29T11:29:08.432 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:08.439+0000 7f5f41ffb640 10 client.?.objecter ms_handle_connect 0x7f5f4c0b0770 2024-03-29T11:29:08.432 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:08.439+0000 7f5f41ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:29:08.432 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:08.440+0000 7f5f41ffb640 1 -- 172.21.15.84:0/3875439917 <== mon.1 v1:172.21.15.179:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 656486914 0 0) 0x7f5f4c125d60 con 0x7f5f4c0b0770 2024-03-29T11:29:08.432 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:08.440+0000 7f5f41ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:29:08.432 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:08.440+0000 7f5f41ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:29:08.432 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:08.440+0000 7f5f41ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:29:08.432 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:08.440+0000 7f5f41ffb640 10 monclient(hunting): my global_id is 4513 2024-03-29T11:29:08.432 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:08.440+0000 7f5f41ffb640 1 -- 172.21.15.84:0/3875439917 --> v1:172.21.15.179:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5f30004eb0 con 0x7f5f4c0b0770 2024-03-29T11:29:08.433 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:08.440+0000 7f5f41ffb640 10 client.?.objecter ms_handle_connect 0x7f5f4c14a700 2024-03-29T11:29:08.433 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:08.440+0000 7f5f41ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:29:08.433 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:08.441+0000 7f5f41ffb640 1 -- 172.21.15.84:0/3875439917 <== mon.1 v1:172.21.15.179:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 175469141 0 0) 0x7f5f30004eb0 con 0x7f5f4c0b0770 2024-03-29T11:29:08.433 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:08.441+0000 7f5f41ffb640 1 -- 172.21.15.84:0/3875439917 --> v1:172.21.15.179:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5f4c125d60 con 0x7f5f4c0b0770 2024-03-29T11:29:08.433 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:08.441+0000 7f5f41ffb640 1 -- 172.21.15.84:0/3875439917 <== mon.1 v1:172.21.15.179:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4236868694 0 0) 0x7f5f3c001990 con 0x7f5f4c0b0770 2024-03-29T11:29:08.433 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:08.441+0000 7f5f41ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:29:08.433 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:08.441+0000 7f5f41ffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:29:08.433 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:08.441+0000 7f5f41ffb640 10 monclient(hunting): dump: 2024-03-29T11:29:08.433 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T11:29:08.433 INFO:teuthology.misc.health.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:29:08.433 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T11:18:53.864025+0000 2024-03-29T11:29:08.433 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T11:18:53.864025+0000 2024-03-29T11:29:08.433 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T11:29:08.433 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:29:08.434 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T11:29:08.434 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T11:29:08.434 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T11:29:08.434 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:29:08.434 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:08.441+0000 7f5f41ffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:29:08.434 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:08.441+0000 7f5f41ffb640 10 client.?.objecter ms_handle_connect 0x7f5f4c0ac5a0 2024-03-29T11:29:08.434 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:08.441+0000 7f5f41ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:29:08.438 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:08.441+0000 7f5f41ffb640 1 -- 172.21.15.84:0/3875439917 <== mon.1 v1:172.21.15.179:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2478702242 0 0) 0x7f5f4c125d60 con 0x7f5f4c0b0770 2024-03-29T11:29:08.438 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:08.441+0000 7f5f41ffb640 1 -- 172.21.15.84:0/3875439917 >> v1:172.21.15.84:6790/0 conn(0x7f5f4c0ac5a0 legacy=0x7f5f4c1deb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:08.438 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:08.441+0000 7f5f41ffb640 1 -- 172.21.15.84:0/3875439917 >> v1:172.21.15.84:6789/0 conn(0x7f5f4c14a700 legacy=0x7f5f4c1e6f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:08.438 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:08.441+0000 7f5f41ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:29:08.438 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:08.441+0000 7f5f41ffb640 1 monclient: found mon.b 2024-03-29T11:29:08.438 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:08.441+0000 7f5f41ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:29:08.438 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:08.441+0000 7f5f41ffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-29T11:29:08.438 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:08.441+0000 7f5f41ffb640 1 -- 172.21.15.84:0/3875439917 --> v1:172.21.15.179:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f4c1258d0 con 0x7f5f4c0b0770 2024-03-29T11:29:08.438 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:08.441+0000 7f5f41ffb640 10 monclient: _finish_auth 0 2024-03-29T11:29:08.438 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:08.441+0000 7f5f41ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:29:08.438 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:08.441+0000 7f5f41ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:29:08.439 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:08.441+0000 7f5f53915640 5 monclient: authenticate success, global_id 4513 2024-03-29T11:29:08.439 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:08.441+0000 7f5f53915640 10 monclient: _renew_subs 2024-03-29T11:29:08.439 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:08.441+0000 7f5f53915640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-29T11:29:08.439 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:08.441+0000 7f5f53915640 1 -- 172.21.15.84:0/3875439917 --> v1:172.21.15.179:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5f4c1df270 con 0x7f5f4c0b0770 2024-03-29T11:29:08.439 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:08.442+0000 7f5f53915640 10 client.4513.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:29:08.439 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:08.442+0000 7f5f53915640 10 monclient: _renew_subs 2024-03-29T11:29:08.439 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:08.442+0000 7f5f53915640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-29T11:29:08.439 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:08.442+0000 7f5f53915640 1 -- 172.21.15.84:0/3875439917 --> v1:172.21.15.179:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f5f4c125520 con 0x7f5f4c0b0770 2024-03-29T11:29:08.439 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:08.442+0000 7f5f53915640 1 librados: init done 2024-03-29T11:29:08.439 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:08.443+0000 7f5f41ffb640 1 -- 172.21.15.84:0/3875439917 <== mon.1 v1:172.21.15.179:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5f3c001c50 con 0x7f5f4c0b0770 2024-03-29T11:29:08.439 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:08.443+0000 7f5f41ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:29:08.439 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:08.443+0000 7f5f41ffb640 1 -- 172.21.15.84:0/3875439917 <== mon.1 v1:172.21.15.179:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4236868694 0 0) 0x7f5f3c003a60 con 0x7f5f4c0b0770 2024-03-29T11:29:08.439 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:08.443+0000 7f5f41ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:29:08.439 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:08.443+0000 7f5f41ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:29:08.439 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:08.443+0000 7f5f41ffb640 10 monclient: dump: 2024-03-29T11:29:08.440 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T11:29:08.440 INFO:teuthology.misc.health.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:29:08.440 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T11:18:53.864025+0000 2024-03-29T11:29:08.440 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T11:18:53.864025+0000 2024-03-29T11:29:08.440 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T11:29:08.440 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:29:08.440 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T11:29:08.440 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T11:29:08.440 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T11:29:08.440 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:29:08.440 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:08.444+0000 7f5f437fe640 4 set_mon_vals no callback set 2024-03-29T11:29:08.440 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:08.444+0000 7f5f53915640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:29:08.440 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:08.446+0000 7f5f41ffb640 1 -- 172.21.15.84:0/3875439917 <== mon.1 v1:172.21.15.179:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4033942909 0 0) 0x7f5f3c003bc0 con 0x7f5f4c0b0770 2024-03-29T11:29:08.440 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:08.445+0000 7f5f53915640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:29:08.441 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:08.446+0000 7f5f53915640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-29T11:29:08.441 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:08.446+0000 7f5f53915640 1 -- 172.21.15.84:0/3875439917 --> v1:172.21.15.179:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5f2c005080 con 0x7f5f4c0b0770 2024-03-29T11:29:08.441 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:08.446+0000 7f5f41ffb640 1 -- 172.21.15.84:0/3875439917 <== mon.1 v1:172.21.15.179:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2808139587 0 0) 0x7f5f3c084ab0 con 0x7f5f4c0b0770 2024-03-29T11:29:08.441 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:08.446+0000 7f5f41ffb640 10 client.4513.objecter ms_dispatch 0x7f5f4c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:29:08.441 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:08.446+0000 7f5f41ffb640 3 client.4513.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:29:08.441 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:08.446+0000 7f5f41ffb640 3 client.4513.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:29:08.441 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:08.446+0000 7f5f41ffb640 20 client.4513.objecter dump_active .. 0 homeless 2024-03-29T11:29:08.441 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:08.447+0000 7f5f516b1640 10 monclient: get_auth_request con 0x7f5f3006dc90 auth_method 0 2024-03-29T11:29:08.441 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:08.448+0000 7f5f41ffb640 10 client.4513.objecter ms_handle_connect 0x7f5f3006dc90 2024-03-29T11:29:08.444 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:08.453+0000 7f5f41ffb640 1 -- 172.21.15.84:0/3875439917 <== mon.1 v1:172.21.15.179:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f5f3c050390 con 0x7f5f4c0b0770 2024-03-29T11:29:08.444 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:08.453+0000 7f5f41ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:29:08.444 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:08.453+0000 7f5f41ffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:29:08.808 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:08.817+0000 7f5f53915640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:29:08.808 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:08.817+0000 7f5f53915640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:29:08.808 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:08.817+0000 7f5f53915640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-29T11:29:08.809 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:08.817+0000 7f5f53915640 1 -- 172.21.15.84:0/3875439917 --> v1:172.21.15.179:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5f2c005330 con 0x7f5f4c0b0770 2024-03-29T11:29:08.809 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:08.818+0000 7f5f41ffb640 1 -- 172.21.15.84:0/3875439917 <== mon.1 v1:172.21.15.179:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4209477549) 0x7f5f3c0522c0 con 0x7f5f4c0b0770 2024-03-29T11:29:08.809 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:08.818+0000 7f5f41ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:29:08.809 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:08.818+0000 7f5f41ffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:29:08.810 INFO:teuthology.misc.health.smithi084.stdout: 2024-03-29T11:29:08.810 INFO:teuthology.misc.health.smithi084.stdout:{"epoch":10,"fsid":"2c3e002f-9eb0-4860-853b-4ef1ef562213","created":"2024-03-29T11:19:32.029169+0000","modified":"2024-03-29T11:19:34.200299+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:34.200299+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"5c39e276-2034-4716-bbc0-57a634eefa92","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"cd580e99-109f-4bf1-88c0-a5987bc8b9ed","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"11e60a5a-051d-453f-a190-66387cb35592","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"2c329fbe-bbc4-4030-bf87-d711cad04679","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f4213d4a-bd36-4433-91ae-d7e2b02481d3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"f0cf44ea-6055-48af-805b-aed858102ff9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"d895eaa0-cc0b-4ee4-9b5a-2dfcd89a60b7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"9bc5a7bc-8f52-4db9-96d6-de5e68d52eaa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:29:08.812 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:08.822+0000 7f5f53915640 10 librados: watch_flush enter 2024-03-29T11:29:08.813 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:08.822+0000 7f5f53915640 10 librados: watch_flush exit 2024-03-29T11:29:08.813 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:08.822+0000 7f5f53915640 20 client.4513.objecter shutdown clearing up homeless session... 2024-03-29T11:29:08.813 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:08.822+0000 7f5f53915640 10 client.4513.objecter successfully canceled tick 2024-03-29T11:29:08.814 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:08.823+0000 7f5f53915640 1 -- 172.21.15.84:0/3875439917 >> v1:172.21.15.179:6816/2981134980 conn(0x7f5f3006dc90 legacy=0x7f5f30070150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:08.814 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:08.823+0000 7f5f53915640 10 monclient: shutdown 2024-03-29T11:29:08.814 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:08.823+0000 7f5f53915640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:29:08.814 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:08.823+0000 7f5f53915640 1 -- 172.21.15.84:0/3875439917 >> v1:172.21.15.179:6789/0 conn(0x7f5f4c0b0770 legacy=0x7f5f4c143dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:08.818 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:08.827+0000 7f5f53915640 1 -- 172.21.15.84:0/3875439917 shutdown_connections 2024-03-29T11:29:08.818 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:08.827+0000 7f5f53915640 1 -- 172.21.15.84:0/3875439917 >> 172.21.15.84:0/3875439917 conn(0x7f5f4c06b660 msgr2=0x7f5f4c0b3e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:29:08.818 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:08.827+0000 7f5f53915640 1 -- 172.21.15.84:0/3875439917 shutdown_connections 2024-03-29T11:29:08.819 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:08.828+0000 7f5f53915640 1 -- 172.21.15.84:0/3875439917 wait complete. 2024-03-29T11:29:08.819 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:08.828+0000 7f5f53915640 1 librados: shutdown 2024-03-29T11:29:08.838 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:29:15.645 DEBUG:teuthology.orchestra.run.smithi084:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:29:15.793 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:15.801+0000 7f8f32d54640 10 monclient: get_monmap_and_config 2024-03-29T11:29:15.793 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:15.801+0000 7f8f32d54640 10 monclient: build_initial_monmap 2024-03-29T11:29:15.794 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:15.801+0000 7f8f32d54640 10 monclient: monmap: 2024-03-29T11:29:15.794 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-29T11:29:15.794 INFO:teuthology.misc.health.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:29:15.794 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T11:29:15.803579+0000 2024-03-29T11:29:15.794 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T11:29:15.803579+0000 2024-03-29T11:29:15.794 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-29T11:29:15.794 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:29:15.794 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-29T11:29:15.794 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.noname-c 2024-03-29T11:29:15.794 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-b 2024-03-29T11:29:15.794 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:29:15.796 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:15.804+0000 7f8f32d54640 1 Processor -- start 2024-03-29T11:29:15.796 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:15.804+0000 7f8f32d54640 1 -- start start 2024-03-29T11:29:15.796 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:15.804+0000 7f8f32d54640 10 monclient: init 2024-03-29T11:29:15.799 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:15.805+0000 7f8f32d54640 10 monclient: _reopen_session rank -1 2024-03-29T11:29:15.800 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:15.805+0000 7f8f32d54640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:29:15.800 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:15.805+0000 7f8f32d54640 10 monclient(hunting): picked mon.noname-a con 0x7f8f240389e0 addr v1:172.21.15.84:6789/0 2024-03-29T11:29:15.800 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:15.805+0000 7f8f32d54640 10 monclient(hunting): picked mon.noname-b con 0x7f8f24038dc0 addr v1:172.21.15.84:6790/0 2024-03-29T11:29:15.800 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:15.808+0000 7f8f30af0640 1 --1- >> v1:172.21.15.84:6789/0 conn(0x7f8f240389e0 0x7f8f24033520 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.84:6789/0 says I am v1:172.21.15.84:53045/0 (socket says 172.21.15.84:53045) 2024-03-29T11:29:15.800 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:15.808+0000 7f8f30af0640 1 -- 172.21.15.84:0/1166673586 learned_addr learned my addr 172.21.15.84:0/1166673586 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-29T11:29:15.801 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:15.808+0000 7f8f32d54640 10 monclient(hunting): picked mon.noname-c con 0x7f8f24030660 addr v1:172.21.15.179:6789/0 2024-03-29T11:29:15.801 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:15.808+0000 7f8f32d54640 1 -- 172.21.15.84:0/1166673586 --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8f240c54c0 con 0x7f8f240389e0 2024-03-29T11:29:15.801 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:15.808+0000 7f8f32d54640 1 -- 172.21.15.84:0/1166673586 --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8f24036430 con 0x7f8f24030660 2024-03-29T11:29:15.801 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:15.808+0000 7f8f32d54640 1 -- 172.21.15.84:0/1166673586 --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8f24039840 con 0x7f8f24038dc0 2024-03-29T11:29:15.801 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:15.808+0000 7f8f32d54640 10 monclient(hunting): _renew_subs 2024-03-29T11:29:15.802 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:15.808+0000 7f8f32d54640 10 monclient(hunting): authenticate will time out at 1417.103873s 2024-03-29T11:29:15.802 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:15.809+0000 7f8f237fe640 1 -- 172.21.15.84:0/1166673586 <== mon.0 v1:172.21.15.84:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3598877624 0 0) 0x7f8f240c54c0 con 0x7f8f240389e0 2024-03-29T11:29:15.802 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:15.809+0000 7f8f237fe640 10 monclient(hunting): _negotiate 2024-03-29T11:29:15.802 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:15.809+0000 7f8f237fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:29:15.802 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:15.809+0000 7f8f237fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:29:15.802 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:15.809+0000 7f8f237fe640 10 monclient(hunting): my global_id is 4569 2024-03-29T11:29:15.802 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:15.809+0000 7f8f237fe640 1 -- 172.21.15.84:0/1166673586 --> v1:172.21.15.84:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8f14003520 con 0x7f8f240389e0 2024-03-29T11:29:15.802 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:15.810+0000 7f8f237fe640 1 -- 172.21.15.84:0/1166673586 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1938656460 0 0) 0x7f8f24039840 con 0x7f8f24038dc0 2024-03-29T11:29:15.803 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:15.810+0000 7f8f237fe640 10 monclient(hunting): _negotiate 2024-03-29T11:29:15.803 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:15.810+0000 7f8f237fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:29:15.803 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:15.810+0000 7f8f237fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:29:15.809 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:15.810+0000 7f8f237fe640 10 monclient(hunting): my global_id is 4631 2024-03-29T11:29:15.809 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:15.810+0000 7f8f237fe640 1 -- 172.21.15.84:0/1166673586 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8f240c54c0 con 0x7f8f24038dc0 2024-03-29T11:29:15.809 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:15.810+0000 7f8f237fe640 1 -- 172.21.15.84:0/1166673586 <== mon.0 v1:172.21.15.84:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1153144187 0 0) 0x7f8f14003520 con 0x7f8f240389e0 2024-03-29T11:29:15.809 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:15.810+0000 7f8f237fe640 1 -- 172.21.15.84:0/1166673586 >> v1:172.21.15.84:6790/0 conn(0x7f8f24038dc0 legacy=0x7f8f240cadd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:15.809 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:15.811+0000 7f8f237fe640 1 -- 172.21.15.84:0/1166673586 >> v1:172.21.15.179:6789/0 conn(0x7f8f24030660 legacy=0x7f8f240d3170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:15.809 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:15.811+0000 7f8f237fe640 10 monclient: _finish_hunting 0 2024-03-29T11:29:15.809 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:15.811+0000 7f8f237fe640 1 monclient: found mon.noname-a 2024-03-29T11:29:15.810 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:15.811+0000 7f8f237fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:29:15.810 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:15.811+0000 7f8f237fe640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.84:6789/0 2024-03-29T11:29:15.810 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:15.811+0000 7f8f237fe640 1 -- 172.21.15.84:0/1166673586 --> v1:172.21.15.84:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f24035690 con 0x7f8f240389e0 2024-03-29T11:29:15.810 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:15.811+0000 7f8f237fe640 10 monclient: _finish_auth 0 2024-03-29T11:29:15.810 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:15.811+0000 7f8f237fe640 10 monclient: _check_auth_tickets 2024-03-29T11:29:15.810 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:15.811+0000 7f8f237fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:29:15.810 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:15.811+0000 7f8f237fe640 1 -- 172.21.15.84:0/1166673586 <== mon.0 v1:172.21.15.84:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4236868694 0 0) 0x7f8f28001690 con 0x7f8f240389e0 2024-03-29T11:29:15.810 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:15.811+0000 7f8f237fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:29:15.810 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:15.811+0000 7f8f237fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:29:15.810 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:15.811+0000 7f8f237fe640 10 monclient: dump: 2024-03-29T11:29:15.810 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T11:29:15.810 INFO:teuthology.misc.health.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:29:15.810 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T11:18:53.864025+0000 2024-03-29T11:29:15.810 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T11:18:53.864025+0000 2024-03-29T11:29:15.811 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T11:29:15.811 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:29:15.811 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T11:29:15.811 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T11:29:15.811 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T11:29:15.811 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:29:15.811 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:15.811+0000 7f8f32d54640 5 monclient: authenticate success, global_id 4569 2024-03-29T11:29:15.811 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:15.811+0000 7f8f32d54640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:29:15.811 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:15.811+0000 7f8f237fe640 1 -- 172.21.15.84:0/1166673586 <== mon.0 v1:172.21.15.84:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8f280017f0 con 0x7f8f240389e0 2024-03-29T11:29:15.811 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:15.811+0000 7f8f237fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:29:15.811 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:15.811+0000 7f8f237fe640 1 -- 172.21.15.84:0/1166673586 <== mon.0 v1:172.21.15.84:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4236868694 0 0) 0x7f8f28001c40 con 0x7f8f240389e0 2024-03-29T11:29:15.811 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:15.811+0000 7f8f237fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:29:15.811 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:15.811+0000 7f8f237fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:29:15.811 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:15.811+0000 7f8f237fe640 10 monclient: dump: 2024-03-29T11:29:15.812 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T11:29:15.812 INFO:teuthology.misc.health.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:29:15.812 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T11:18:53.864025+0000 2024-03-29T11:29:15.812 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T11:18:53.864025+0000 2024-03-29T11:29:15.812 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T11:29:15.812 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:29:15.812 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T11:29:15.812 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T11:29:15.812 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T11:29:15.812 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:29:15.812 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:15.811+0000 7f8f32d54640 10 monclient: get_monmap_and_config success 2024-03-29T11:29:15.812 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:15.811+0000 7f8f32d54640 4 set_mon_vals no callback set 2024-03-29T11:29:15.812 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:15.812+0000 7f8f32d54640 10 monclient: shutdown 2024-03-29T11:29:15.812 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:15.812+0000 7f8f32d54640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:29:15.813 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:15.812+0000 7f8f32d54640 1 -- 172.21.15.84:0/1166673586 >> v1:172.21.15.84:6789/0 conn(0x7f8f240389e0 legacy=0x7f8f24033520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:15.813 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:15.812+0000 7f8f32d54640 1 -- 172.21.15.84:0/1166673586 shutdown_connections 2024-03-29T11:29:15.813 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:15.812+0000 7f8f32d54640 1 -- 172.21.15.84:0/1166673586 >> 172.21.15.84:0/1166673586 conn(0x7f8f24005c00 msgr2=0x7f8f24006010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:29:15.813 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:15.812+0000 7f8f32d54640 1 -- 172.21.15.84:0/1166673586 shutdown_connections 2024-03-29T11:29:15.813 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:15.812+0000 7f8f32d54640 1 -- 172.21.15.84:0/1166673586 wait complete. 2024-03-29T11:29:15.813 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:15.813+0000 7f8f32d54640 10 monclient: build_initial_monmap 2024-03-29T11:29:15.813 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:15.813+0000 7f8f32d54640 10 monclient: monmap: 2024-03-29T11:29:15.813 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-29T11:29:15.813 INFO:teuthology.misc.health.smithi084.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:29:15.813 INFO:teuthology.misc.health.smithi084.stderr:last_changed 0.000000 2024-03-29T11:29:15.813 INFO:teuthology.misc.health.smithi084.stderr:created 0.000000 2024-03-29T11:29:15.813 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-29T11:29:15.813 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:29:15.813 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-29T11:29:15.813 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.noname-b 2024-03-29T11:29:15.814 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-c 2024-03-29T11:29:15.814 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:29:15.814 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:15.814+0000 7f8f32d54640 1 librados: starting msgr at 2024-03-29T11:29:15.814 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:15.814+0000 7f8f32d54640 1 librados: starting objecter 2024-03-29T11:29:15.814 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:15.814+0000 7f8f32d54640 1 Processor -- start 2024-03-29T11:29:15.814 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:15.814+0000 7f8f32d54640 1 -- start start 2024-03-29T11:29:15.814 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:15.814+0000 7f8f32d54640 1 librados: setting wanted keys 2024-03-29T11:29:15.814 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:15.814+0000 7f8f32d54640 1 librados: calling monclient init 2024-03-29T11:29:15.814 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:15.814+0000 7f8f32d54640 10 monclient: init 2024-03-29T11:29:15.814 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:15.815+0000 7f8f32d54640 10 monclient: _reopen_session rank -1 2024-03-29T11:29:15.814 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:15.815+0000 7f8f32d54640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:29:15.821 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:15.817+0000 7f8f32d54640 10 monclient(hunting): picked mon.noname-b con 0x7f8f24030660 addr v1:172.21.15.179:6789/0 2024-03-29T11:29:15.821 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:15.817+0000 7f8f32d54640 10 monclient(hunting): picked mon.noname-c con 0x7f8f240389e0 addr v1:172.21.15.84:6790/0 2024-03-29T11:29:15.821 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:15.817+0000 7f8f23fff640 1 --1- >> v1:172.21.15.84:6790/0 conn(0x7f8f240389e0 0x7f8f241600d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.84:6790/0 says I am v1:172.21.15.84:45168/0 (socket says 172.21.15.84:45168) 2024-03-29T11:29:15.821 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:15.817+0000 7f8f23fff640 1 -- 172.21.15.84:0/97746814 learned_addr learned my addr 172.21.15.84:0/97746814 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-29T11:29:15.821 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:15.817+0000 7f8f32d54640 10 monclient(hunting): picked mon.noname-a con 0x7f8f241607e0 addr v1:172.21.15.84:6789/0 2024-03-29T11:29:15.822 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:15.817+0000 7f8f32d54640 1 -- 172.21.15.84:0/97746814 --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8f240c5eb0 con 0x7f8f241607e0 2024-03-29T11:29:15.822 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:15.817+0000 7f8f32d54640 1 -- 172.21.15.84:0/97746814 --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8f240c5b60 con 0x7f8f24030660 2024-03-29T11:29:15.822 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:15.817+0000 7f8f217fa640 10 client.?.objecter ms_handle_connect 0x7f8f240389e0 2024-03-29T11:29:15.822 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:15.817+0000 7f8f217fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:29:15.822 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:15.817+0000 7f8f32d54640 1 -- 172.21.15.84:0/97746814 --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8f2415a4b0 con 0x7f8f240389e0 2024-03-29T11:29:15.822 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:15.817+0000 7f8f32d54640 10 monclient(hunting): _renew_subs 2024-03-29T11:29:15.822 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:15.817+0000 7f8f32d54640 10 monclient(hunting): authenticate will time out at 1417.112716s 2024-03-29T11:29:15.822 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:15.818+0000 7f8f217fa640 1 -- 172.21.15.84:0/97746814 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1877496908 0 0) 0x7f8f2415a4b0 con 0x7f8f240389e0 2024-03-29T11:29:15.822 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:15.818+0000 7f8f217fa640 10 monclient(hunting): _negotiate 2024-03-29T11:29:15.823 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:15.818+0000 7f8f217fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:29:15.823 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:15.818+0000 7f8f217fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:29:15.823 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:15.818+0000 7f8f217fa640 10 monclient(hunting): my global_id is 4634 2024-03-29T11:29:15.823 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:15.818+0000 7f8f217fa640 1 -- 172.21.15.84:0/97746814 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8f18004eb0 con 0x7f8f240389e0 2024-03-29T11:29:15.823 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:15.819+0000 7f8f217fa640 1 -- 172.21.15.84:0/97746814 <== mon.2 v1:172.21.15.84:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1984704856 0 0) 0x7f8f18004eb0 con 0x7f8f240389e0 2024-03-29T11:29:15.823 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:15.820+0000 7f8f217fa640 1 -- 172.21.15.84:0/97746814 --> v1:172.21.15.84:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8f2415a4b0 con 0x7f8f240389e0 2024-03-29T11:29:15.823 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:15.820+0000 7f8f217fa640 1 -- 172.21.15.84:0/97746814 <== mon.2 v1:172.21.15.84:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4236868694 0 0) 0x7f8f1c001920 con 0x7f8f240389e0 2024-03-29T11:29:15.823 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:15.820+0000 7f8f217fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:29:15.823 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:15.820+0000 7f8f217fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:29:15.823 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:15.820+0000 7f8f217fa640 10 monclient(hunting): dump: 2024-03-29T11:29:15.824 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T11:29:15.824 INFO:teuthology.misc.health.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:29:15.824 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T11:18:53.864025+0000 2024-03-29T11:29:15.824 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T11:18:53.864025+0000 2024-03-29T11:29:15.824 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T11:29:15.824 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:29:15.824 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T11:29:15.824 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T11:29:15.824 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T11:29:15.824 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:29:15.824 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:15.820+0000 7f8f217fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:29:15.824 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:15.820+0000 7f8f217fa640 1 -- 172.21.15.84:0/97746814 <== mon.2 v1:172.21.15.84:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2829374633 0 0) 0x7f8f2415a4b0 con 0x7f8f240389e0 2024-03-29T11:29:15.824 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:15.820+0000 7f8f217fa640 1 -- 172.21.15.84:0/97746814 >> v1:172.21.15.179:6789/0 conn(0x7f8f24030660 legacy=0x7f8f2402fa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:15.824 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:15.820+0000 7f8f217fa640 1 -- 172.21.15.84:0/97746814 >> v1:172.21.15.84:6789/0 conn(0x7f8f241607e0 legacy=0x7f8f24167490 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:29:15.825 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:15.822+0000 7f8f217fa640 10 monclient: _finish_hunting 0 2024-03-29T11:29:15.825 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:15.822+0000 7f8f217fa640 1 monclient: found mon.c 2024-03-29T11:29:15.825 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:15.822+0000 7f8f217fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:29:15.825 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:15.822+0000 7f8f217fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-29T11:29:15.825 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:15.822+0000 7f8f217fa640 1 -- 172.21.15.84:0/97746814 --> v1:172.21.15.84:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f24160c70 con 0x7f8f240389e0 2024-03-29T11:29:15.825 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:15.822+0000 7f8f217fa640 10 monclient: _finish_auth 0 2024-03-29T11:29:15.825 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:15.822+0000 7f8f217fa640 10 monclient: _check_auth_tickets 2024-03-29T11:29:15.825 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:15.822+0000 7f8f217fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:29:15.825 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:15.822+0000 7f8f32d54640 5 monclient: authenticate success, global_id 4634 2024-03-29T11:29:15.825 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:15.822+0000 7f8f32d54640 10 monclient: _renew_subs 2024-03-29T11:29:15.825 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:15.822+0000 7f8f32d54640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-29T11:29:15.825 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:15.822+0000 7f8f32d54640 1 -- 172.21.15.84:0/97746814 --> v1:172.21.15.84:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8f240c61f0 con 0x7f8f240389e0 2024-03-29T11:29:15.825 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:15.822+0000 7f8f32d54640 10 client.4634.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:29:15.825 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:15.822+0000 7f8f32d54640 10 monclient: _renew_subs 2024-03-29T11:29:15.825 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:15.822+0000 7f8f32d54640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-29T11:29:15.826 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:15.822+0000 7f8f32d54640 1 -- 172.21.15.84:0/97746814 --> v1:172.21.15.84:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f8f24168d90 con 0x7f8f240389e0 2024-03-29T11:29:15.826 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:15.822+0000 7f8f32d54640 1 librados: init done 2024-03-29T11:29:15.826 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:15.822+0000 7f8f217fa640 1 -- 172.21.15.84:0/97746814 <== mon.2 v1:172.21.15.84:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8f1c001d70 con 0x7f8f240389e0 2024-03-29T11:29:15.826 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:15.822+0000 7f8f217fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:29:15.826 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:15.822+0000 7f8f217fa640 1 -- 172.21.15.84:0/97746814 <== mon.2 v1:172.21.15.84:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4236868694 0 0) 0x7f8f1c003a40 con 0x7f8f240389e0 2024-03-29T11:29:15.826 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:15.822+0000 7f8f217fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:29:15.826 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:15.822+0000 7f8f217fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:29:15.826 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:15.822+0000 7f8f217fa640 10 monclient: dump: 2024-03-29T11:29:15.826 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T11:29:15.826 INFO:teuthology.misc.health.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:29:15.826 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T11:18:53.864025+0000 2024-03-29T11:29:15.826 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T11:18:53.864025+0000 2024-03-29T11:29:15.826 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T11:29:15.826 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:29:15.827 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T11:29:15.827 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T11:29:15.827 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T11:29:15.827 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:29:15.827 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:15.822+0000 7f8f227fc640 4 set_mon_vals no callback set 2024-03-29T11:29:15.827 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:15.824+0000 7f8f12ffd640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:29:15.827 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:15.824+0000 7f8f217fa640 1 -- 172.21.15.84:0/97746814 <== mon.2 v1:172.21.15.84:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4033942909 0 0) 0x7f8f1c003ba0 con 0x7f8f240389e0 2024-03-29T11:29:15.827 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:15.825+0000 7f8f217fa640 1 -- 172.21.15.84:0/97746814 <== mon.2 v1:172.21.15.84:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2808139587 0 0) 0x7f8f1c084aa0 con 0x7f8f240389e0 2024-03-29T11:29:15.827 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:15.825+0000 7f8f217fa640 10 client.4634.objecter ms_dispatch 0x7f8f2c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:29:15.827 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:15.825+0000 7f8f217fa640 3 client.4634.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:29:15.827 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:15.825+0000 7f8f217fa640 3 client.4634.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:29:15.827 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:15.825+0000 7f8f217fa640 20 client.4634.objecter dump_active .. 0 homeless 2024-03-29T11:29:15.827 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:15.826+0000 7f8f12ffd640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:29:15.827 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:15.826+0000 7f8f12ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-29T11:29:15.828 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:15.826+0000 7f8f12ffd640 1 -- 172.21.15.84:0/97746814 --> v1:172.21.15.84:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8f24007b90 con 0x7f8f240389e0 2024-03-29T11:29:15.828 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:15.826+0000 7f8f30af0640 10 monclient: get_auth_request con 0x7f8f1806dc90 auth_method 0 2024-03-29T11:29:15.828 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:15.826+0000 7f8f217fa640 10 client.4634.objecter ms_handle_connect 0x7f8f1806dc90 2024-03-29T11:29:15.828 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:15.836+0000 7f8f217fa640 1 -- 172.21.15.84:0/97746814 <== mon.2 v1:172.21.15.84:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f8f1c050380 con 0x7f8f240389e0 2024-03-29T11:29:15.828 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:15.836+0000 7f8f217fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:29:15.828 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:15.836+0000 7f8f217fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:29:16.147 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:16.155+0000 7f8f12ffd640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:29:16.147 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:16.155+0000 7f8f12ffd640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:29:16.147 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:16.155+0000 7f8f12ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-29T11:29:16.147 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:16.155+0000 7f8f12ffd640 1 -- 172.21.15.84:0/97746814 --> v1:172.21.15.84:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8f24005120 con 0x7f8f240389e0 2024-03-29T11:29:16.151 INFO:teuthology.misc.health.smithi084.stdout: 2024-03-29T11:29:16.151 INFO:teuthology.misc.health.smithi084.stdout:{"epoch":10,"fsid":"2c3e002f-9eb0-4860-853b-4ef1ef562213","created":"2024-03-29T11:19:32.029169+0000","modified":"2024-03-29T11:19:34.200299+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:34.200299+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"5c39e276-2034-4716-bbc0-57a634eefa92","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"cd580e99-109f-4bf1-88c0-a5987bc8b9ed","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"11e60a5a-051d-453f-a190-66387cb35592","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"2c329fbe-bbc4-4030-bf87-d711cad04679","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f4213d4a-bd36-4433-91ae-d7e2b02481d3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"f0cf44ea-6055-48af-805b-aed858102ff9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"d895eaa0-cc0b-4ee4-9b5a-2dfcd89a60b7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"9bc5a7bc-8f52-4db9-96d6-de5e68d52eaa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:29:16.152 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:16.156+0000 7f8f217fa640 1 -- 172.21.15.84:0/97746814 <== mon.2 v1:172.21.15.84:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4209477549) 0x7f8f1c0522b0 con 0x7f8f240389e0 2024-03-29T11:29:16.152 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:16.156+0000 7f8f217fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:29:16.152 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:16.156+0000 7f8f217fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:29:16.152 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:16.160+0000 7f8f12ffd640 10 librados: watch_flush enter 2024-03-29T11:29:16.152 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:16.160+0000 7f8f12ffd640 10 librados: watch_flush exit 2024-03-29T11:29:16.153 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:16.160+0000 7f8f12ffd640 20 client.4634.objecter shutdown clearing up homeless session... 2024-03-29T11:29:16.153 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:16.160+0000 7f8f12ffd640 10 client.4634.objecter successfully canceled tick 2024-03-29T11:29:16.153 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:16.160+0000 7f8f12ffd640 1 -- 172.21.15.84:0/97746814 >> v1:172.21.15.179:6816/2981134980 conn(0x7f8f1806dc90 legacy=0x7f8f18070150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:16.153 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:16.160+0000 7f8f12ffd640 10 monclient: shutdown 2024-03-29T11:29:16.153 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:16.160+0000 7f8f12ffd640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:29:16.153 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:16.160+0000 7f8f12ffd640 1 -- 172.21.15.84:0/97746814 >> v1:172.21.15.84:6790/0 conn(0x7f8f240389e0 legacy=0x7f8f241600d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:16.153 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:16.160+0000 7f8f12ffd640 1 -- 172.21.15.84:0/97746814 shutdown_connections 2024-03-29T11:29:16.153 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:16.160+0000 7f8f12ffd640 1 -- 172.21.15.84:0/97746814 >> 172.21.15.84:0/97746814 conn(0x7f8f24005c00 msgr2=0x7f8f24034090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:29:16.154 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:16.163+0000 7f8f12ffd640 1 -- 172.21.15.84:0/97746814 shutdown_connections 2024-03-29T11:29:16.155 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:16.163+0000 7f8f12ffd640 1 -- 172.21.15.84:0/97746814 wait complete. 2024-03-29T11:29:16.155 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:16.163+0000 7f8f12ffd640 1 librados: shutdown 2024-03-29T11:29:16.172 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:29:22.979 DEBUG:teuthology.orchestra.run.smithi084:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:29:23.123 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:23.131+0000 7fda82579640 10 monclient: get_monmap_and_config 2024-03-29T11:29:23.123 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:23.131+0000 7fda82579640 10 monclient: build_initial_monmap 2024-03-29T11:29:23.123 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:23.131+0000 7fda82579640 10 monclient: monmap: 2024-03-29T11:29:23.123 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-29T11:29:23.123 INFO:teuthology.misc.health.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:29:23.123 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T11:29:23.132484+0000 2024-03-29T11:29:23.123 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T11:29:23.132484+0000 2024-03-29T11:29:23.123 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-29T11:29:23.123 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:29:23.123 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-29T11:29:23.123 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.noname-c 2024-03-29T11:29:23.123 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-b 2024-03-29T11:29:23.124 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:29:23.124 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:23.132+0000 7fda82579640 1 Processor -- start 2024-03-29T11:29:23.124 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:23.132+0000 7fda82579640 1 -- start start 2024-03-29T11:29:23.124 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:23.132+0000 7fda82579640 10 monclient: init 2024-03-29T11:29:23.124 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:23.133+0000 7fda82579640 10 monclient: _reopen_session rank -1 2024-03-29T11:29:23.124 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:23.133+0000 7fda82579640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:29:23.125 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:23.133+0000 7fda82579640 10 monclient(hunting): picked mon.noname-c con 0x7fda7c0ad850 addr v1:172.21.15.179:6789/0 2024-03-29T11:29:23.125 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:23.133+0000 7fda82579640 10 monclient(hunting): picked mon.noname-a con 0x7fda7c0b0770 addr v1:172.21.15.84:6789/0 2024-03-29T11:29:23.125 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:23.133+0000 7fda82579640 10 monclient(hunting): picked mon.noname-b con 0x7fda7c13c1b0 addr v1:172.21.15.84:6790/0 2024-03-29T11:29:23.125 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:23.133+0000 7fda82579640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fda7c0ac8e0 con 0x7fda7c0b0770 2024-03-29T11:29:23.125 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:23.133+0000 7fda82579640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fda7c0ad330 con 0x7fda7c0ad850 2024-03-29T11:29:23.125 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:23.133+0000 7fda82579640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fda7c0acfe0 con 0x7fda7c13c1b0 2024-03-29T11:29:23.125 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:23.133+0000 7fda82579640 10 monclient(hunting): _renew_subs 2024-03-29T11:29:23.125 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:23.133+0000 7fda82579640 10 monclient(hunting): authenticate will time out at 1424.427828s 2024-03-29T11:29:23.126 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:23.134+0000 7fda80b16640 1 --1- >> v1:172.21.15.84:6790/0 conn(0x7fda7c13c1b0 0x7fda7c14b960 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.84:6790/0 says I am v1:172.21.15.84:37852/0 (socket says 172.21.15.84:37852) 2024-03-29T11:29:23.126 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:23.134+0000 7fda80b16640 1 -- 172.21.15.84:0/4281252070 learned_addr learned my addr 172.21.15.84:0/4281252070 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-29T11:29:23.126 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:23.134+0000 7fda7b7fe640 1 --1- >> v1:172.21.15.84:6789/0 conn(0x7fda7c0b0770 0x7fda7c0aaae0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.84:6789/0 says I am v1:172.21.15.84:55162/0 (socket says 172.21.15.84:55162) 2024-03-29T11:29:23.127 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:23.134+0000 7fda7affd640 1 -- 172.21.15.84:0/4281252070 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1792548868 0 0) 0x7fda7c0acfe0 con 0x7fda7c13c1b0 2024-03-29T11:29:23.127 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:23.134+0000 7fda7affd640 10 monclient(hunting): _negotiate 2024-03-29T11:29:23.127 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:23.134+0000 7fda7affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:29:23.127 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:23.134+0000 7fda7affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:29:23.127 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:23.134+0000 7fda7affd640 10 monclient(hunting): my global_id is 4637 2024-03-29T11:29:23.127 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:23.134+0000 7fda7affd640 1 -- 172.21.15.84:0/4281252070 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fda64003520 con 0x7fda7c13c1b0 2024-03-29T11:29:23.127 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:23.134+0000 7fda7affd640 1 -- 172.21.15.84:0/4281252070 <== mon.1 v1:172.21.15.179:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1561314679 0 0) 0x7fda7c0ad330 con 0x7fda7c0ad850 2024-03-29T11:29:23.127 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:23.134+0000 7fda7affd640 10 monclient(hunting): _negotiate 2024-03-29T11:29:23.127 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:23.134+0000 7fda7affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:29:23.127 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:23.134+0000 7fda7affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:29:23.127 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:23.134+0000 7fda7affd640 10 monclient(hunting): my global_id is 4519 2024-03-29T11:29:23.127 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:23.134+0000 7fda7affd640 1 -- 172.21.15.84:0/4281252070 --> v1:172.21.15.179:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fda7c0acfe0 con 0x7fda7c0ad850 2024-03-29T11:29:23.127 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:23.134+0000 7fda7affd640 1 -- 172.21.15.84:0/4281252070 <== mon.2 v1:172.21.15.84:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3935208175 0 0) 0x7fda64003520 con 0x7fda7c13c1b0 2024-03-29T11:29:23.127 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:23.134+0000 7fda7affd640 1 -- 172.21.15.84:0/4281252070 >> v1:172.21.15.179:6789/0 conn(0x7fda7c0ad850 legacy=0x7fda7c072070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:23.128 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:23.134+0000 7fda7affd640 1 -- 172.21.15.84:0/4281252070 >> v1:172.21.15.84:6789/0 conn(0x7fda7c0b0770 legacy=0x7fda7c0aaae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:23.128 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:23.134+0000 7fda7affd640 10 monclient: _finish_hunting 0 2024-03-29T11:29:23.128 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:23.134+0000 7fda7affd640 1 monclient: found mon.noname-b 2024-03-29T11:29:23.128 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:23.134+0000 7fda7affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:29:23.128 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:23.134+0000 7fda7affd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.84:6790/0 2024-03-29T11:29:23.128 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:23.134+0000 7fda7affd640 1 -- 172.21.15.84:0/4281252070 --> v1:172.21.15.84:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda7c0ab1f0 con 0x7fda7c13c1b0 2024-03-29T11:29:23.128 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:23.134+0000 7fda7affd640 10 monclient: _finish_auth 0 2024-03-29T11:29:23.128 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:23.134+0000 7fda7affd640 10 monclient: _check_auth_tickets 2024-03-29T11:29:23.128 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:23.134+0000 7fda7affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:29:23.128 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:23.134+0000 7fda7affd640 1 -- 172.21.15.84:0/4281252070 <== mon.2 v1:172.21.15.84:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4236868694 0 0) 0x7fda6c001690 con 0x7fda7c13c1b0 2024-03-29T11:29:23.128 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:23.134+0000 7fda7affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:29:23.128 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:23.134+0000 7fda7affd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:29:23.128 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:23.134+0000 7fda7affd640 10 monclient: dump: 2024-03-29T11:29:23.128 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T11:29:23.128 INFO:teuthology.misc.health.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:29:23.129 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T11:18:53.864025+0000 2024-03-29T11:29:23.129 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T11:18:53.864025+0000 2024-03-29T11:29:23.129 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T11:29:23.129 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:29:23.129 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T11:29:23.129 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T11:29:23.129 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T11:29:23.129 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:29:23.129 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:23.134+0000 7fda82579640 5 monclient: authenticate success, global_id 4637 2024-03-29T11:29:23.129 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:23.134+0000 7fda82579640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:29:23.129 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:23.135+0000 7fda7affd640 1 -- 172.21.15.84:0/4281252070 <== mon.2 v1:172.21.15.84:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fda6c0018b0 con 0x7fda7c13c1b0 2024-03-29T11:29:23.129 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:23.135+0000 7fda7affd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:29:23.129 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:23.135+0000 7fda82579640 10 monclient: get_monmap_and_config success 2024-03-29T11:29:23.129 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:23.135+0000 7fda82579640 4 set_mon_vals no callback set 2024-03-29T11:29:23.130 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:23.135+0000 7fda7affd640 1 -- 172.21.15.84:0/4281252070 <== mon.2 v1:172.21.15.84:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4236868694 0 0) 0x7fda6c001c40 con 0x7fda7c13c1b0 2024-03-29T11:29:23.130 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:23.135+0000 7fda7affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:29:23.130 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:23.135+0000 7fda7affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:29:23.130 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:23.135+0000 7fda7affd640 10 monclient: dump: 2024-03-29T11:29:23.130 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T11:29:23.130 INFO:teuthology.misc.health.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:29:23.130 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T11:18:53.864025+0000 2024-03-29T11:29:23.130 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T11:18:53.864025+0000 2024-03-29T11:29:23.130 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T11:29:23.130 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:29:23.131 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T11:29:23.131 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T11:29:23.131 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T11:29:23.131 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:29:23.140 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:23.136+0000 7fda82579640 10 monclient: shutdown 2024-03-29T11:29:23.140 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:23.136+0000 7fda82579640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:29:23.140 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:23.136+0000 7fda82579640 1 -- 172.21.15.84:0/4281252070 >> v1:172.21.15.84:6790/0 conn(0x7fda7c13c1b0 legacy=0x7fda7c14b960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:23.140 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:23.136+0000 7fda82579640 1 -- 172.21.15.84:0/4281252070 shutdown_connections 2024-03-29T11:29:23.140 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:23.136+0000 7fda82579640 1 -- 172.21.15.84:0/4281252070 >> 172.21.15.84:0/4281252070 conn(0x7fda7c06b660 msgr2=0x7fda7c06ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:29:23.140 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:23.136+0000 7fda82579640 1 -- 172.21.15.84:0/4281252070 shutdown_connections 2024-03-29T11:29:23.140 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:23.136+0000 7fda82579640 1 -- 172.21.15.84:0/4281252070 wait complete. 2024-03-29T11:29:23.140 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:23.137+0000 7fda82579640 10 monclient: build_initial_monmap 2024-03-29T11:29:23.141 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:23.137+0000 7fda82579640 10 monclient: monmap: 2024-03-29T11:29:23.141 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-29T11:29:23.141 INFO:teuthology.misc.health.smithi084.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:29:23.141 INFO:teuthology.misc.health.smithi084.stderr:last_changed 0.000000 2024-03-29T11:29:23.141 INFO:teuthology.misc.health.smithi084.stderr:created 0.000000 2024-03-29T11:29:23.141 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-29T11:29:23.141 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:29:23.141 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-29T11:29:23.141 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.noname-b 2024-03-29T11:29:23.141 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-c 2024-03-29T11:29:23.141 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:29:23.141 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:23.137+0000 7fda82579640 1 librados: starting msgr at 2024-03-29T11:29:23.141 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:23.137+0000 7fda82579640 1 librados: starting objecter 2024-03-29T11:29:23.141 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:23.137+0000 7fda82579640 1 Processor -- start 2024-03-29T11:29:23.141 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:23.137+0000 7fda82579640 1 -- start start 2024-03-29T11:29:23.142 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:23.137+0000 7fda82579640 1 librados: setting wanted keys 2024-03-29T11:29:23.142 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:23.137+0000 7fda82579640 1 librados: calling monclient init 2024-03-29T11:29:23.142 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:23.137+0000 7fda82579640 10 monclient: init 2024-03-29T11:29:23.142 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:23.138+0000 7fda82579640 10 monclient: _reopen_session rank -1 2024-03-29T11:29:23.142 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:23.138+0000 7fda82579640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:29:23.142 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:23.138+0000 7fda82579640 10 monclient(hunting): picked mon.noname-a con 0x7fda7c0ad850 addr v1:172.21.15.84:6789/0 2024-03-29T11:29:23.142 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:23.138+0000 7fda82579640 10 monclient(hunting): picked mon.noname-c con 0x7fda7c0b0770 addr v1:172.21.15.84:6790/0 2024-03-29T11:29:23.142 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:23.138+0000 7fda82579640 10 monclient(hunting): picked mon.noname-b con 0x7fda7c13c1b0 addr v1:172.21.15.179:6789/0 2024-03-29T11:29:23.142 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:23.138+0000 7fda82579640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fda7c13fe50 con 0x7fda7c0ad850 2024-03-29T11:29:23.142 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:23.138+0000 7fda82579640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fda7c13cd00 con 0x7fda7c13c1b0 2024-03-29T11:29:23.142 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:23.138+0000 7fda82579640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fda7c13e650 con 0x7fda7c0b0770 2024-03-29T11:29:23.142 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:23.138+0000 7fda82579640 10 monclient(hunting): _renew_subs 2024-03-29T11:29:23.142 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:23.138+0000 7fda82579640 10 monclient(hunting): authenticate will time out at 1424.432992s 2024-03-29T11:29:23.143 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:23.139+0000 7fda7b7fe640 1 --1- >> v1:172.21.15.84:6790/0 conn(0x7fda7c0b0770 0x7fda7c13f740 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.84:6790/0 says I am v1:172.21.15.84:37960/0 (socket says 172.21.15.84:37960) 2024-03-29T11:29:23.143 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:23.139+0000 7fda80b16640 1 --1- >> v1:172.21.15.179:6789/0 conn(0x7fda7c13c1b0 0x7fda7c1e6020 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.179:6789/0 says I am v1:172.21.15.84:51146/0 (socket says 172.21.15.84:51146) 2024-03-29T11:29:23.143 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:23.139+0000 7fda7b7fe640 1 -- 172.21.15.84:0/3662877756 learned_addr learned my addr 172.21.15.84:0/3662877756 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-29T11:29:23.143 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:23.139+0000 7fda78ff9640 10 client.?.objecter ms_handle_connect 0x7fda7c13c1b0 2024-03-29T11:29:23.143 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:23.139+0000 7fda78ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:29:23.143 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:23.139+0000 7fda78ff9640 10 client.?.objecter ms_handle_connect 0x7fda7c0b0770 2024-03-29T11:29:23.143 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:23.139+0000 7fda78ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:29:23.143 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:23.139+0000 7fda78ff9640 10 client.?.objecter ms_handle_connect 0x7fda7c0ad850 2024-03-29T11:29:23.143 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:23.139+0000 7fda78ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:29:23.143 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:23.139+0000 7fda78ff9640 1 -- 172.21.15.84:0/3662877756 <== mon.1 v1:172.21.15.179:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2471832087 0 0) 0x7fda7c13cd00 con 0x7fda7c13c1b0 2024-03-29T11:29:23.143 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:23.139+0000 7fda78ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:29:23.143 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:23.139+0000 7fda78ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:29:23.143 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:23.139+0000 7fda78ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:29:23.143 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:23.139+0000 7fda78ff9640 10 monclient(hunting): my global_id is 4522 2024-03-29T11:29:23.143 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:23.139+0000 7fda78ff9640 1 -- 172.21.15.84:0/3662877756 --> v1:172.21.15.179:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fda68004eb0 con 0x7fda7c13c1b0 2024-03-29T11:29:23.144 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:23.139+0000 7fda78ff9640 1 -- 172.21.15.84:0/3662877756 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 602410122 0 0) 0x7fda7c13e650 con 0x7fda7c0b0770 2024-03-29T11:29:23.144 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:23.139+0000 7fda78ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:29:23.144 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:23.139+0000 7fda78ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:29:23.144 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:23.139+0000 7fda78ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:29:23.144 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:23.139+0000 7fda78ff9640 10 monclient(hunting): my global_id is 4640 2024-03-29T11:29:23.144 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:23.139+0000 7fda78ff9640 1 -- 172.21.15.84:0/3662877756 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fda7c13cd00 con 0x7fda7c0b0770 2024-03-29T11:29:23.144 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:23.139+0000 7fda78ff9640 1 -- 172.21.15.84:0/3662877756 <== mon.0 v1:172.21.15.84:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 629953879 0 0) 0x7fda7c13fe50 con 0x7fda7c0ad850 2024-03-29T11:29:23.144 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:23.139+0000 7fda78ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:29:23.144 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:23.139+0000 7fda78ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:29:23.144 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:23.139+0000 7fda78ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:29:23.144 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:23.139+0000 7fda78ff9640 10 monclient(hunting): my global_id is 4575 2024-03-29T11:29:23.144 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:23.139+0000 7fda78ff9640 1 -- 172.21.15.84:0/3662877756 --> v1:172.21.15.84:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fda7c13e650 con 0x7fda7c0ad850 2024-03-29T11:29:23.144 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:23.140+0000 7fda78ff9640 1 -- 172.21.15.84:0/3662877756 <== mon.1 v1:172.21.15.179:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 714093688 0 0) 0x7fda68004eb0 con 0x7fda7c13c1b0 2024-03-29T11:29:23.144 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:23.140+0000 7fda78ff9640 1 -- 172.21.15.84:0/3662877756 --> v1:172.21.15.179:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fda7c13fe50 con 0x7fda7c13c1b0 2024-03-29T11:29:23.144 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:23.140+0000 7fda78ff9640 1 -- 172.21.15.84:0/3662877756 <== mon.2 v1:172.21.15.84:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2455529202 0 0) 0x7fda7c13cd00 con 0x7fda7c0b0770 2024-03-29T11:29:23.145 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:23.140+0000 7fda78ff9640 1 -- 172.21.15.84:0/3662877756 --> v1:172.21.15.84:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fda68005a30 con 0x7fda7c0b0770 2024-03-29T11:29:23.145 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:23.140+0000 7fda78ff9640 1 -- 172.21.15.84:0/3662877756 <== mon.2 v1:172.21.15.84:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4236868694 0 0) 0x7fda70001930 con 0x7fda7c0b0770 2024-03-29T11:29:23.145 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:23.140+0000 7fda78ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:29:23.145 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:23.140+0000 7fda78ff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:29:23.145 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:23.140+0000 7fda78ff9640 10 monclient(hunting): dump: 2024-03-29T11:29:23.145 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T11:29:23.145 INFO:teuthology.misc.health.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:29:23.145 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T11:18:53.864025+0000 2024-03-29T11:29:23.145 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T11:18:53.864025+0000 2024-03-29T11:29:23.145 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T11:29:23.145 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:29:23.145 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T11:29:23.145 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T11:29:23.145 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T11:29:23.145 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:29:23.146 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:23.140+0000 7fda78ff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:29:23.146 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:23.140+0000 7fda78ff9640 1 -- 172.21.15.84:0/3662877756 <== mon.1 v1:172.21.15.179:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4236868694 0 0) 0x7fda6c001bc0 con 0x7fda7c13c1b0 2024-03-29T11:29:23.146 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:23.140+0000 7fda78ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:29:23.146 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:23.140+0000 7fda78ff9640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:29:23.146 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:23.140+0000 7fda78ff9640 10 monclient(hunting): dump: 2024-03-29T11:29:23.146 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T11:29:23.146 INFO:teuthology.misc.health.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:29:23.146 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T11:18:53.864025+0000 2024-03-29T11:29:23.146 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T11:18:53.864025+0000 2024-03-29T11:29:23.146 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T11:29:23.146 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:29:23.146 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T11:29:23.146 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T11:29:23.146 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T11:29:23.147 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:29:23.147 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:23.140+0000 7fda78ff9640 1 -- 172.21.15.84:0/3662877756 <== mon.0 v1:172.21.15.84:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2624570205 0 0) 0x7fda7c13e650 con 0x7fda7c0ad850 2024-03-29T11:29:23.147 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:23.140+0000 7fda78ff9640 1 -- 172.21.15.84:0/3662877756 --> v1:172.21.15.84:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fda7c13cd00 con 0x7fda7c0ad850 2024-03-29T11:29:23.147 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:23.140+0000 7fda78ff9640 1 -- 172.21.15.84:0/3662877756 <== mon.0 v1:172.21.15.84:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4236868694 0 0) 0x7fda74001cb0 con 0x7fda7c0ad850 2024-03-29T11:29:23.147 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:23.140+0000 7fda78ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:29:23.147 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:23.140+0000 7fda78ff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:29:23.147 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:23.140+0000 7fda78ff9640 10 monclient(hunting): dump: 2024-03-29T11:29:23.147 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T11:29:23.147 INFO:teuthology.misc.health.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:29:23.147 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T11:18:53.864025+0000 2024-03-29T11:29:23.147 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T11:18:53.864025+0000 2024-03-29T11:29:23.147 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T11:29:23.147 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:29:23.147 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T11:29:23.147 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T11:29:23.148 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T11:29:23.148 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:29:23.148 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:23.141+0000 7fda78ff9640 1 -- 172.21.15.84:0/3662877756 <== mon.2 v1:172.21.15.84:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3243280484 0 0) 0x7fda68005a30 con 0x7fda7c0b0770 2024-03-29T11:29:23.148 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:23.141+0000 7fda78ff9640 1 -- 172.21.15.84:0/3662877756 >> v1:172.21.15.179:6789/0 conn(0x7fda7c13c1b0 legacy=0x7fda7c1e6020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:23.148 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:23.141+0000 7fda78ff9640 1 -- 172.21.15.84:0/3662877756 >> v1:172.21.15.84:6789/0 conn(0x7fda7c0ad850 legacy=0x7fda7c13f030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:23.148 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:23.141+0000 7fda78ff9640 10 monclient: _finish_hunting 0 2024-03-29T11:29:23.148 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:23.141+0000 7fda78ff9640 1 monclient: found mon.c 2024-03-29T11:29:23.148 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:23.141+0000 7fda78ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:29:23.148 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:23.141+0000 7fda78ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-29T11:29:23.148 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:23.141+0000 7fda78ff9640 1 -- 172.21.15.84:0/3662877756 --> v1:172.21.15.84:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda7c13e930 con 0x7fda7c0b0770 2024-03-29T11:29:23.148 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:23.141+0000 7fda78ff9640 10 monclient: _finish_auth 0 2024-03-29T11:29:23.148 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:23.141+0000 7fda78ff9640 10 monclient: _check_auth_tickets 2024-03-29T11:29:23.148 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:23.141+0000 7fda78ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:29:23.148 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:23.141+0000 7fda82579640 5 monclient: authenticate success, global_id 4640 2024-03-29T11:29:23.149 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:23.141+0000 7fda82579640 10 monclient: _renew_subs 2024-03-29T11:29:23.149 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:23.141+0000 7fda82579640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-29T11:29:23.149 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:23.141+0000 7fda82579640 1 -- 172.21.15.84:0/3662877756 --> v1:172.21.15.84:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fda7c13ec10 con 0x7fda7c0b0770 2024-03-29T11:29:23.149 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:23.141+0000 7fda82579640 10 client.4640.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:29:23.149 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:23.141+0000 7fda82579640 10 monclient: _renew_subs 2024-03-29T11:29:23.149 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:23.141+0000 7fda82579640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-29T11:29:23.149 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:23.141+0000 7fda82579640 1 -- 172.21.15.84:0/3662877756 --> v1:172.21.15.84:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fda7c145350 con 0x7fda7c0b0770 2024-03-29T11:29:23.149 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:23.141+0000 7fda82579640 1 librados: init done 2024-03-29T11:29:23.149 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:23.141+0000 7fda78ff9640 1 -- 172.21.15.84:0/3662877756 <== mon.2 v1:172.21.15.84:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fda700047e0 con 0x7fda7c0b0770 2024-03-29T11:29:23.149 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:23.141+0000 7fda78ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:29:23.149 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:23.141+0000 7fda78ff9640 1 -- 172.21.15.84:0/3662877756 <== mon.2 v1:172.21.15.84:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4236868694 0 0) 0x7fda70004af0 con 0x7fda7c0b0770 2024-03-29T11:29:23.149 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:23.141+0000 7fda78ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:29:23.149 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:23.142+0000 7fda78ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:29:23.149 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:23.142+0000 7fda78ff9640 10 monclient: dump: 2024-03-29T11:29:23.150 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T11:29:23.150 INFO:teuthology.misc.health.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:29:23.150 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T11:18:53.864025+0000 2024-03-29T11:29:23.150 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T11:18:53.864025+0000 2024-03-29T11:29:23.150 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T11:29:23.150 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:29:23.150 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T11:29:23.150 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T11:29:23.150 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T11:29:23.150 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:29:23.150 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:23.142+0000 7fda7a7fc640 4 set_mon_vals no callback set 2024-03-29T11:29:23.150 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:23.142+0000 7fda82579640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:29:23.150 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:23.142+0000 7fda82579640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:29:23.150 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:23.142+0000 7fda82579640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-29T11:29:23.151 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:23.142+0000 7fda82579640 1 -- 172.21.15.84:0/3662877756 --> v1:172.21.15.84:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fda5c005080 con 0x7fda7c0b0770 2024-03-29T11:29:23.151 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:23.144+0000 7fda78ff9640 1 -- 172.21.15.84:0/3662877756 <== mon.2 v1:172.21.15.84:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4033942909 0 0) 0x7fda7001aac0 con 0x7fda7c0b0770 2024-03-29T11:29:23.151 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:23.145+0000 7fda78ff9640 1 -- 172.21.15.84:0/3662877756 <== mon.2 v1:172.21.15.84:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2808139587 0 0) 0x7fda70084970 con 0x7fda7c0b0770 2024-03-29T11:29:23.151 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:23.145+0000 7fda78ff9640 10 client.4640.objecter ms_dispatch 0x7fda7c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:29:23.151 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:23.145+0000 7fda78ff9640 3 client.4640.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:29:23.151 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:23.145+0000 7fda78ff9640 3 client.4640.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:29:23.151 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:23.145+0000 7fda78ff9640 20 client.4640.objecter dump_active .. 0 homeless 2024-03-29T11:29:23.151 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:23.145+0000 7fda7bfff640 10 monclient: get_auth_request con 0x7fda6806e470 auth_method 0 2024-03-29T11:29:23.151 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:23.145+0000 7fda78ff9640 10 client.4640.objecter ms_handle_connect 0x7fda6806e470 2024-03-29T11:29:23.152 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:23.157+0000 7fda78ff9640 1 -- 172.21.15.84:0/3662877756 <== mon.2 v1:172.21.15.84:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fda70050250 con 0x7fda7c0b0770 2024-03-29T11:29:23.152 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:23.157+0000 7fda78ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:29:23.152 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:23.157+0000 7fda78ff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:29:23.502 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:23.511+0000 7fda82579640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:29:23.502 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:23.511+0000 7fda82579640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:29:23.503 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:23.511+0000 7fda82579640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-29T11:29:23.503 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:23.511+0000 7fda82579640 1 -- 172.21.15.84:0/3662877756 --> v1:172.21.15.84:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fda5c005330 con 0x7fda7c0b0770 2024-03-29T11:29:23.503 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:23.512+0000 7fda78ff9640 1 -- 172.21.15.84:0/3662877756 <== mon.2 v1:172.21.15.84:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4209477549) 0x7fda70052180 con 0x7fda7c0b0770 2024-03-29T11:29:23.503 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:23.512+0000 7fda78ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:29:23.503 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:23.512+0000 7fda78ff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:29:23.504 INFO:teuthology.misc.health.smithi084.stdout: 2024-03-29T11:29:23.504 INFO:teuthology.misc.health.smithi084.stdout:{"epoch":10,"fsid":"2c3e002f-9eb0-4860-853b-4ef1ef562213","created":"2024-03-29T11:19:32.029169+0000","modified":"2024-03-29T11:19:34.200299+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:34.200299+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"5c39e276-2034-4716-bbc0-57a634eefa92","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"cd580e99-109f-4bf1-88c0-a5987bc8b9ed","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"11e60a5a-051d-453f-a190-66387cb35592","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"2c329fbe-bbc4-4030-bf87-d711cad04679","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f4213d4a-bd36-4433-91ae-d7e2b02481d3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"f0cf44ea-6055-48af-805b-aed858102ff9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"d895eaa0-cc0b-4ee4-9b5a-2dfcd89a60b7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"9bc5a7bc-8f52-4db9-96d6-de5e68d52eaa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:29:23.507 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:23.515+0000 7fda82579640 10 librados: watch_flush enter 2024-03-29T11:29:23.507 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:23.515+0000 7fda82579640 10 librados: watch_flush exit 2024-03-29T11:29:23.507 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:23.515+0000 7fda82579640 20 client.4640.objecter shutdown clearing up homeless session... 2024-03-29T11:29:23.507 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:23.515+0000 7fda82579640 10 client.4640.objecter successfully canceled tick 2024-03-29T11:29:23.507 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:23.516+0000 7fda82579640 1 -- 172.21.15.84:0/3662877756 >> v1:172.21.15.179:6816/2981134980 conn(0x7fda6806e470 legacy=0x7fda68070930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:23.507 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:23.516+0000 7fda82579640 10 monclient: shutdown 2024-03-29T11:29:23.507 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:23.516+0000 7fda82579640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:29:23.507 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:23.516+0000 7fda82579640 1 -- 172.21.15.84:0/3662877756 >> v1:172.21.15.84:6790/0 conn(0x7fda7c0b0770 legacy=0x7fda7c13f740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:23.507 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:23.516+0000 7fda82579640 1 -- 172.21.15.84:0/3662877756 shutdown_connections 2024-03-29T11:29:23.507 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:23.516+0000 7fda82579640 1 -- 172.21.15.84:0/3662877756 >> 172.21.15.84:0/3662877756 conn(0x7fda7c06b660 msgr2=0x7fda7c133ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:29:23.508 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:23.516+0000 7fda82579640 1 -- 172.21.15.84:0/3662877756 shutdown_connections 2024-03-29T11:29:23.508 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:23.516+0000 7fda82579640 1 -- 172.21.15.84:0/3662877756 wait complete. 2024-03-29T11:29:23.508 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:23.516+0000 7fda82579640 1 librados: shutdown 2024-03-29T11:29:23.521 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:29:30.329 DEBUG:teuthology.orchestra.run.smithi084:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:29:30.463 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:30.470+0000 7f2d9556c640 10 monclient: get_monmap_and_config 2024-03-29T11:29:30.463 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:30.470+0000 7f2d9556c640 10 monclient: build_initial_monmap 2024-03-29T11:29:30.463 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:30.470+0000 7f2d9556c640 10 monclient: monmap: 2024-03-29T11:29:30.463 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-29T11:29:30.463 INFO:teuthology.misc.health.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:29:30.463 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T11:29:30.471363+0000 2024-03-29T11:29:30.463 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T11:29:30.471363+0000 2024-03-29T11:29:30.463 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-29T11:29:30.463 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:29:30.463 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-29T11:29:30.463 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.noname-c 2024-03-29T11:29:30.463 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-b 2024-03-29T11:29:30.464 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:29:30.464 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:30.471+0000 7f2d9556c640 1 Processor -- start 2024-03-29T11:29:30.464 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:30.471+0000 7f2d9556c640 1 -- start start 2024-03-29T11:29:30.464 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:30.471+0000 7f2d9556c640 10 monclient: init 2024-03-29T11:29:30.466 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:30.472+0000 7f2d9556c640 10 monclient: _reopen_session rank -1 2024-03-29T11:29:30.466 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:30.472+0000 7f2d9556c640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:29:30.466 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:30.472+0000 7f2d9556c640 10 monclient(hunting): picked mon.noname-c con 0x7f2d901457f0 addr v1:172.21.15.179:6789/0 2024-03-29T11:29:30.466 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:30.472+0000 7f2d9556c640 10 monclient(hunting): picked mon.noname-b con 0x7f2d9013f650 addr v1:172.21.15.84:6790/0 2024-03-29T11:29:30.467 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:30.472+0000 7f2d9556c640 10 monclient(hunting): picked mon.noname-a con 0x7f2d900a9d70 addr v1:172.21.15.84:6789/0 2024-03-29T11:29:30.467 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:30.472+0000 7f2d9556c640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2d900aaef0 con 0x7f2d900a9d70 2024-03-29T11:29:30.467 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:30.472+0000 7f2d9556c640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2d90145bd0 con 0x7f2d901457f0 2024-03-29T11:29:30.467 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:30.472+0000 7f2d9556c640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2d90145eb0 con 0x7f2d9013f650 2024-03-29T11:29:30.467 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:30.472+0000 7f2d9556c640 10 monclient(hunting): _renew_subs 2024-03-29T11:29:30.467 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:30.472+0000 7f2d9556c640 10 monclient(hunting): authenticate will time out at 1431.766658s 2024-03-29T11:29:30.467 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:30.472+0000 7f2d8e7fc640 1 --1- >> v1:172.21.15.84:6790/0 conn(0x7f2d9013f650 0x7f2d900b2620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.84:6790/0 says I am v1:172.21.15.84:39260/0 (socket says 172.21.15.84:39260) 2024-03-29T11:29:30.467 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:30.472+0000 7f2d8e7fc640 1 -- 172.21.15.84:0/2988705505 learned_addr learned my addr 172.21.15.84:0/2988705505 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-29T11:29:30.467 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:30.473+0000 7f2d8dffb640 1 -- 172.21.15.84:0/2988705505 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 755561775 0 0) 0x7f2d90145eb0 con 0x7f2d9013f650 2024-03-29T11:29:30.467 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:30.473+0000 7f2d8dffb640 10 monclient(hunting): _negotiate 2024-03-29T11:29:30.467 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:30.473+0000 7f2d8dffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:29:30.468 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:30.473+0000 7f2d8dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:29:30.468 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:30.473+0000 7f2d8dffb640 10 monclient(hunting): my global_id is 4643 2024-03-29T11:29:30.468 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:30.473+0000 7f2d8dffb640 1 -- 172.21.15.84:0/2988705505 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2d78003520 con 0x7f2d9013f650 2024-03-29T11:29:30.468 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:30.473+0000 7f2d8dffb640 1 -- 172.21.15.84:0/2988705505 <== mon.2 v1:172.21.15.84:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1777762426 0 0) 0x7f2d78003520 con 0x7f2d9013f650 2024-03-29T11:29:30.468 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:30.473+0000 7f2d8dffb640 1 -- 172.21.15.84:0/2988705505 >> v1:172.21.15.179:6789/0 conn(0x7f2d901457f0 legacy=0x7f2d9013ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:30.468 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:30.473+0000 7f2d8dffb640 1 -- 172.21.15.84:0/2988705505 >> v1:172.21.15.84:6789/0 conn(0x7f2d900a9d70 legacy=0x7f2d9014d8c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:29:30.468 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:30.474+0000 7f2d8dffb640 10 monclient: _finish_hunting 0 2024-03-29T11:29:30.468 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:30.474+0000 7f2d8dffb640 1 monclient: found mon.noname-b 2024-03-29T11:29:30.468 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:30.474+0000 7f2d8dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:29:30.468 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:30.474+0000 7f2d8dffb640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.84:6790/0 2024-03-29T11:29:30.468 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:30.474+0000 7f2d8dffb640 1 -- 172.21.15.84:0/2988705505 --> v1:172.21.15.84:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d90146340 con 0x7f2d9013f650 2024-03-29T11:29:30.468 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:30.474+0000 7f2d8dffb640 10 monclient: _finish_auth 0 2024-03-29T11:29:30.468 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:30.474+0000 7f2d8dffb640 10 monclient: _check_auth_tickets 2024-03-29T11:29:30.468 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:30.474+0000 7f2d8dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:29:30.469 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:30.474+0000 7f2d9556c640 5 monclient: authenticate success, global_id 4643 2024-03-29T11:29:30.469 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:30.474+0000 7f2d9556c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:29:30.469 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:30.474+0000 7f2d8dffb640 1 -- 172.21.15.84:0/2988705505 <== mon.2 v1:172.21.15.84:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4236868694 0 0) 0x7f2d80001690 con 0x7f2d9013f650 2024-03-29T11:29:30.469 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:30.474+0000 7f2d8dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:29:30.469 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:30.475+0000 7f2d8dffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:29:30.469 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:30.475+0000 7f2d8dffb640 10 monclient: dump: 2024-03-29T11:29:30.469 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T11:29:30.469 INFO:teuthology.misc.health.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:29:30.469 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T11:18:53.864025+0000 2024-03-29T11:29:30.469 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T11:18:53.864025+0000 2024-03-29T11:29:30.469 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T11:29:30.469 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:29:30.469 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T11:29:30.470 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T11:29:30.470 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T11:29:30.470 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:29:30.470 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:30.475+0000 7f2d9556c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:29:30.479 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:30.475+0000 7f2d8dffb640 1 -- 172.21.15.84:0/2988705505 <== mon.2 v1:172.21.15.84:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2d80001920 con 0x7f2d9013f650 2024-03-29T11:29:30.479 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:30.475+0000 7f2d8dffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:29:30.479 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:30.476+0000 7f2d8dffb640 1 -- 172.21.15.84:0/2988705505 <== mon.2 v1:172.21.15.84:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4236868694 0 0) 0x7f2d80001d30 con 0x7f2d9013f650 2024-03-29T11:29:30.479 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:30.476+0000 7f2d8dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:29:30.479 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:30.476+0000 7f2d8dffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:29:30.479 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:30.476+0000 7f2d8dffb640 10 monclient: dump: 2024-03-29T11:29:30.479 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T11:29:30.479 INFO:teuthology.misc.health.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:29:30.479 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T11:18:53.864025+0000 2024-03-29T11:29:30.480 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T11:18:53.864025+0000 2024-03-29T11:29:30.480 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T11:29:30.480 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:29:30.480 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T11:29:30.480 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T11:29:30.480 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T11:29:30.480 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:29:30.480 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:30.476+0000 7f2d9556c640 10 monclient: get_monmap_and_config success 2024-03-29T11:29:30.480 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:30.476+0000 7f2d9556c640 4 set_mon_vals no callback set 2024-03-29T11:29:30.480 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:30.476+0000 7f2d9556c640 10 monclient: shutdown 2024-03-29T11:29:30.480 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:30.476+0000 7f2d9556c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:29:30.480 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:30.476+0000 7f2d9556c640 1 -- 172.21.15.84:0/2988705505 >> v1:172.21.15.84:6790/0 conn(0x7f2d9013f650 legacy=0x7f2d900b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:30.480 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:30.477+0000 7f2d9556c640 1 -- 172.21.15.84:0/2988705505 shutdown_connections 2024-03-29T11:29:30.481 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:30.477+0000 7f2d9556c640 1 -- 172.21.15.84:0/2988705505 >> 172.21.15.84:0/2988705505 conn(0x7f2d90066a00 msgr2=0x7f2d90066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:29:30.481 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:30.477+0000 7f2d9556c640 1 -- 172.21.15.84:0/2988705505 shutdown_connections 2024-03-29T11:29:30.481 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:30.477+0000 7f2d9556c640 1 -- 172.21.15.84:0/2988705505 wait complete. 2024-03-29T11:29:30.481 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:30.477+0000 7f2d9556c640 10 monclient: build_initial_monmap 2024-03-29T11:29:30.481 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:30.477+0000 7f2d9556c640 10 monclient: monmap: 2024-03-29T11:29:30.481 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-29T11:29:30.481 INFO:teuthology.misc.health.smithi084.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:29:30.481 INFO:teuthology.misc.health.smithi084.stderr:last_changed 0.000000 2024-03-29T11:29:30.481 INFO:teuthology.misc.health.smithi084.stderr:created 0.000000 2024-03-29T11:29:30.481 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-29T11:29:30.481 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:29:30.481 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-29T11:29:30.481 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.noname-b 2024-03-29T11:29:30.481 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-c 2024-03-29T11:29:30.481 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:29:30.482 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:30.478+0000 7f2d9556c640 1 librados: starting msgr at 2024-03-29T11:29:30.482 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:30.478+0000 7f2d9556c640 1 librados: starting objecter 2024-03-29T11:29:30.482 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:30.478+0000 7f2d9556c640 1 Processor -- start 2024-03-29T11:29:30.482 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:30.478+0000 7f2d9556c640 1 -- start start 2024-03-29T11:29:30.482 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:30.478+0000 7f2d9556c640 1 librados: setting wanted keys 2024-03-29T11:29:30.482 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:30.478+0000 7f2d9556c640 1 librados: calling monclient init 2024-03-29T11:29:30.482 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:30.478+0000 7f2d9556c640 10 monclient: init 2024-03-29T11:29:30.482 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:30.480+0000 7f2d9556c640 10 monclient: _reopen_session rank -1 2024-03-29T11:29:30.482 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:30.480+0000 7f2d9556c640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:29:30.482 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:30.480+0000 7f2d9556c640 10 monclient(hunting): picked mon.noname-a con 0x7f2d900a9d70 addr v1:172.21.15.84:6789/0 2024-03-29T11:29:30.483 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:30.480+0000 7f2d9556c640 10 monclient(hunting): picked mon.noname-b con 0x7f2d9013f650 addr v1:172.21.15.179:6789/0 2024-03-29T11:29:30.485 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:30.480+0000 7f2d9556c640 10 monclient(hunting): picked mon.noname-c con 0x7f2d901457f0 addr v1:172.21.15.84:6790/0 2024-03-29T11:29:30.485 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:30.480+0000 7f2d9556c640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2d901dfc00 con 0x7f2d900a9d70 2024-03-29T11:29:30.485 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:30.480+0000 7f2d9556c640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2d9006ec50 con 0x7f2d9013f650 2024-03-29T11:29:30.486 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:30.480+0000 7f2d9556c640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2d9006ef30 con 0x7f2d901457f0 2024-03-29T11:29:30.486 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:30.480+0000 7f2d9556c640 10 monclient(hunting): _renew_subs 2024-03-29T11:29:30.486 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:30.480+0000 7f2d9556c640 10 monclient(hunting): authenticate will time out at 1431.774670s 2024-03-29T11:29:30.486 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:30.480+0000 7f2d8e7fc640 1 --1- >> v1:172.21.15.179:6789/0 conn(0x7f2d9013f650 0x7f2d901dede0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.179:6789/0 says I am v1:172.21.15.84:44252/0 (socket says 172.21.15.84:44252) 2024-03-29T11:29:30.486 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:30.480+0000 7f2d8e7fc640 1 -- 172.21.15.84:0/3784633345 learned_addr learned my addr 172.21.15.84:0/3784633345 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-29T11:29:30.486 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:30.480+0000 7f2d777fe640 10 client.?.objecter ms_handle_connect 0x7f2d9013f650 2024-03-29T11:29:30.486 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:30.480+0000 7f2d777fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:29:30.486 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:30.481+0000 7f2d777fe640 10 client.?.objecter ms_handle_connect 0x7f2d901457f0 2024-03-29T11:29:30.486 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:30.481+0000 7f2d777fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:29:30.486 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:30.481+0000 7f2d777fe640 1 -- 172.21.15.84:0/3784633345 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1281045726 0 0) 0x7f2d9006ef30 con 0x7f2d901457f0 2024-03-29T11:29:30.486 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:30.481+0000 7f2d777fe640 10 monclient(hunting): _negotiate 2024-03-29T11:29:30.486 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:30.481+0000 7f2d777fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:29:30.487 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:30.481+0000 7f2d777fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:29:30.487 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:30.481+0000 7f2d777fe640 10 monclient(hunting): my global_id is 4646 2024-03-29T11:29:30.487 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:30.481+0000 7f2d777fe640 1 -- 172.21.15.84:0/3784633345 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2d7c004eb0 con 0x7f2d901457f0 2024-03-29T11:29:30.487 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:30.481+0000 7f2d777fe640 1 -- 172.21.15.84:0/3784633345 <== mon.2 v1:172.21.15.84:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1658346986 0 0) 0x7f2d7c004eb0 con 0x7f2d901457f0 2024-03-29T11:29:30.487 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:30.481+0000 7f2d777fe640 1 -- 172.21.15.84:0/3784633345 --> v1:172.21.15.84:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2d9006ef30 con 0x7f2d901457f0 2024-03-29T11:29:30.487 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:30.481+0000 7f2d777fe640 1 -- 172.21.15.84:0/3784633345 <== mon.2 v1:172.21.15.84:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4236868694 0 0) 0x7f2d88001b70 con 0x7f2d901457f0 2024-03-29T11:29:30.487 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:30.481+0000 7f2d777fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:29:30.487 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:30.481+0000 7f2d777fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:29:30.487 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:30.481+0000 7f2d777fe640 10 monclient(hunting): dump: 2024-03-29T11:29:30.487 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T11:29:30.487 INFO:teuthology.misc.health.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:29:30.487 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T11:18:53.864025+0000 2024-03-29T11:29:30.487 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T11:18:53.864025+0000 2024-03-29T11:29:30.487 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T11:29:30.487 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:29:30.488 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T11:29:30.488 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T11:29:30.488 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T11:29:30.488 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:29:30.488 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:30.481+0000 7f2d777fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:29:30.488 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:30.481+0000 7f2d777fe640 1 -- 172.21.15.84:0/3784633345 <== mon.2 v1:172.21.15.84:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1798934567 0 0) 0x7f2d9006ef30 con 0x7f2d901457f0 2024-03-29T11:29:30.488 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:30.481+0000 7f2d777fe640 1 -- 172.21.15.84:0/3784633345 >> v1:172.21.15.179:6789/0 conn(0x7f2d9013f650 legacy=0x7f2d901dede0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:30.488 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:30.481+0000 7f2d777fe640 1 -- 172.21.15.84:0/3784633345 >> v1:172.21.15.84:6789/0 conn(0x7f2d900a9d70 legacy=0x7f2d90141d50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:29:30.488 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:30.482+0000 7f2d777fe640 10 monclient: _finish_hunting 0 2024-03-29T11:29:30.488 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:30.482+0000 7f2d777fe640 1 monclient: found mon.c 2024-03-29T11:29:30.488 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:30.482+0000 7f2d777fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:29:30.488 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:30.482+0000 7f2d777fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-29T11:29:30.488 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:30.482+0000 7f2d777fe640 1 -- 172.21.15.84:0/3784633345 --> v1:172.21.15.84:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d9006e010 con 0x7f2d901457f0 2024-03-29T11:29:30.488 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:30.482+0000 7f2d777fe640 10 monclient: _finish_auth 0 2024-03-29T11:29:30.489 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:30.482+0000 7f2d777fe640 10 monclient: _check_auth_tickets 2024-03-29T11:29:30.489 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:30.482+0000 7f2d777fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:29:30.489 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:30.482+0000 7f2d9556c640 5 monclient: authenticate success, global_id 4646 2024-03-29T11:29:30.489 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:30.482+0000 7f2d9556c640 10 monclient: _renew_subs 2024-03-29T11:29:30.489 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:30.482+0000 7f2d9556c640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-29T11:29:30.489 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:30.482+0000 7f2d9556c640 1 -- 172.21.15.84:0/3784633345 --> v1:172.21.15.84:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2d90071f90 con 0x7f2d901457f0 2024-03-29T11:29:30.489 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:30.482+0000 7f2d9556c640 10 client.4646.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:29:30.489 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:30.482+0000 7f2d9556c640 10 monclient: _renew_subs 2024-03-29T11:29:30.489 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:30.482+0000 7f2d9556c640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-29T11:29:30.489 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:30.482+0000 7f2d9556c640 1 -- 172.21.15.84:0/3784633345 --> v1:172.21.15.84:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f2d90144c20 con 0x7f2d901457f0 2024-03-29T11:29:30.489 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:30.482+0000 7f2d9556c640 1 librados: init done 2024-03-29T11:29:30.489 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:30.482+0000 7f2d777fe640 1 -- 172.21.15.84:0/3784633345 <== mon.2 v1:172.21.15.84:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2d88003970 con 0x7f2d901457f0 2024-03-29T11:29:30.489 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:30.483+0000 7f2d777fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:29:30.489 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:30.483+0000 7f2d777fe640 1 -- 172.21.15.84:0/3784633345 <== mon.2 v1:172.21.15.84:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4236868694 0 0) 0x7f2d88002100 con 0x7f2d901457f0 2024-03-29T11:29:30.489 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:30.483+0000 7f2d777fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:29:30.490 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:30.483+0000 7f2d777fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:29:30.490 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:30.483+0000 7f2d777fe640 10 monclient: dump: 2024-03-29T11:29:30.490 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T11:29:30.490 INFO:teuthology.misc.health.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:29:30.490 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T11:18:53.864025+0000 2024-03-29T11:29:30.490 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T11:18:53.864025+0000 2024-03-29T11:29:30.490 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T11:29:30.490 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:29:30.490 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T11:29:30.490 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T11:29:30.490 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T11:29:30.490 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:29:30.490 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:30.483+0000 7f2d8d7fa640 4 set_mon_vals no callback set 2024-03-29T11:29:30.490 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:30.484+0000 7f2d9556c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:29:30.490 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:30.484+0000 7f2d9556c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:29:30.491 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:30.484+0000 7f2d9556c640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-29T11:29:30.491 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:30.484+0000 7f2d9556c640 1 -- 172.21.15.84:0/3784633345 --> v1:172.21.15.84:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2d70005080 con 0x7f2d901457f0 2024-03-29T11:29:30.491 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:30.485+0000 7f2d777fe640 1 -- 172.21.15.84:0/3784633345 <== mon.2 v1:172.21.15.84:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4033942909 0 0) 0x7f2d88019af0 con 0x7f2d901457f0 2024-03-29T11:29:30.491 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:30.486+0000 7f2d777fe640 1 -- 172.21.15.84:0/3784633345 <== mon.2 v1:172.21.15.84:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2808139587 0 0) 0x7f2d88083bb0 con 0x7f2d901457f0 2024-03-29T11:29:30.491 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:30.486+0000 7f2d777fe640 10 client.4646.objecter ms_dispatch 0x7f2d90000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:29:30.491 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:30.486+0000 7f2d777fe640 3 client.4646.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:29:30.491 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:30.486+0000 7f2d777fe640 3 client.4646.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:29:30.491 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:30.486+0000 7f2d777fe640 20 client.4646.objecter dump_active .. 0 homeless 2024-03-29T11:29:30.491 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:30.487+0000 7f2d8effd640 10 monclient: get_auth_request con 0x7f2d7c06dc60 auth_method 0 2024-03-29T11:29:30.492 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:30.489+0000 7f2d777fe640 10 client.4646.objecter ms_handle_connect 0x7f2d7c06dc60 2024-03-29T11:29:30.492 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:30.491+0000 7f2d777fe640 1 -- 172.21.15.84:0/3784633345 <== mon.2 v1:172.21.15.84:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f2d8804f490 con 0x7f2d901457f0 2024-03-29T11:29:30.492 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:30.491+0000 7f2d777fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:29:30.492 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:30.491+0000 7f2d777fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:29:30.781 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:30.790+0000 7f2d9556c640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:29:30.782 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:30.790+0000 7f2d9556c640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:29:30.782 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:30.790+0000 7f2d9556c640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-29T11:29:30.782 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:30.790+0000 7f2d9556c640 1 -- 172.21.15.84:0/3784633345 --> v1:172.21.15.84:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2d70005330 con 0x7f2d901457f0 2024-03-29T11:29:30.783 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:30.791+0000 7f2d777fe640 1 -- 172.21.15.84:0/3784633345 <== mon.2 v1:172.21.15.84:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4209477549) 0x7f2d880513c0 con 0x7f2d901457f0 2024-03-29T11:29:30.783 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:30.791+0000 7f2d777fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:29:30.783 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:30.791+0000 7f2d777fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:29:30.783 INFO:teuthology.misc.health.smithi084.stdout: 2024-03-29T11:29:30.783 INFO:teuthology.misc.health.smithi084.stdout:{"epoch":10,"fsid":"2c3e002f-9eb0-4860-853b-4ef1ef562213","created":"2024-03-29T11:19:32.029169+0000","modified":"2024-03-29T11:19:34.200299+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:34.200299+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"5c39e276-2034-4716-bbc0-57a634eefa92","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"cd580e99-109f-4bf1-88c0-a5987bc8b9ed","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"11e60a5a-051d-453f-a190-66387cb35592","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"2c329fbe-bbc4-4030-bf87-d711cad04679","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f4213d4a-bd36-4433-91ae-d7e2b02481d3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"f0cf44ea-6055-48af-805b-aed858102ff9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"d895eaa0-cc0b-4ee4-9b5a-2dfcd89a60b7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"9bc5a7bc-8f52-4db9-96d6-de5e68d52eaa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:29:30.786 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:30.794+0000 7f2d9556c640 10 librados: watch_flush enter 2024-03-29T11:29:30.786 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:30.794+0000 7f2d9556c640 10 librados: watch_flush exit 2024-03-29T11:29:30.786 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:30.794+0000 7f2d9556c640 20 client.4646.objecter shutdown clearing up homeless session... 2024-03-29T11:29:30.786 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:30.794+0000 7f2d9556c640 10 client.4646.objecter successfully canceled tick 2024-03-29T11:29:30.786 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:30.795+0000 7f2d9556c640 1 -- 172.21.15.84:0/3784633345 >> v1:172.21.15.179:6816/2981134980 conn(0x7f2d7c06dc60 legacy=0x7f2d7c070120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:30.786 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:30.795+0000 7f2d9556c640 10 monclient: shutdown 2024-03-29T11:29:30.786 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:30.795+0000 7f2d9556c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:29:30.787 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:30.795+0000 7f2d9556c640 1 -- 172.21.15.84:0/3784633345 >> v1:172.21.15.84:6790/0 conn(0x7f2d901457f0 legacy=0x7f2d901df4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:30.787 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:30.795+0000 7f2d9556c640 1 -- 172.21.15.84:0/3784633345 shutdown_connections 2024-03-29T11:29:30.787 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:30.795+0000 7f2d9556c640 1 -- 172.21.15.84:0/3784633345 >> 172.21.15.84:0/3784633345 conn(0x7f2d90066a00 msgr2=0x7f2d900b5810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:29:30.787 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:30.795+0000 7f2d9556c640 1 -- 172.21.15.84:0/3784633345 shutdown_connections 2024-03-29T11:29:30.787 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:30.796+0000 7f2d9556c640 1 -- 172.21.15.84:0/3784633345 wait complete. 2024-03-29T11:29:30.787 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:30.796+0000 7f2d9556c640 1 librados: shutdown 2024-03-29T11:29:30.800 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:29:37.608 DEBUG:teuthology.orchestra.run.smithi084:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:29:37.754 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:37.763+0000 7ff3b8f23640 10 monclient: get_monmap_and_config 2024-03-29T11:29:37.755 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:37.763+0000 7ff3b8f23640 10 monclient: build_initial_monmap 2024-03-29T11:29:37.755 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:37.763+0000 7ff3b8f23640 10 monclient: monmap: 2024-03-29T11:29:37.755 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-29T11:29:37.755 INFO:teuthology.misc.health.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:29:37.755 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T11:29:37.764259+0000 2024-03-29T11:29:37.755 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T11:29:37.764259+0000 2024-03-29T11:29:37.755 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-29T11:29:37.755 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:29:37.755 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-29T11:29:37.755 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.noname-c 2024-03-29T11:29:37.755 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-b 2024-03-29T11:29:37.755 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:29:37.758 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:37.764+0000 7ff3b8f23640 1 Processor -- start 2024-03-29T11:29:37.758 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:37.764+0000 7ff3b8f23640 1 -- start start 2024-03-29T11:29:37.758 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:37.764+0000 7ff3b8f23640 10 monclient: init 2024-03-29T11:29:37.758 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:37.765+0000 7ff3b8f23640 10 monclient: _reopen_session rank -1 2024-03-29T11:29:37.758 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:37.765+0000 7ff3b8f23640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:29:37.758 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:37.765+0000 7ff3b8f23640 10 monclient(hunting): picked mon.noname-c con 0x7ff3b40b1f60 addr v1:172.21.15.179:6789/0 2024-03-29T11:29:37.758 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:37.765+0000 7ff3b8f23640 10 monclient(hunting): picked mon.noname-b con 0x7ff3b4071eb0 addr v1:172.21.15.84:6790/0 2024-03-29T11:29:37.758 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:37.765+0000 7ff3b8f23640 10 monclient(hunting): picked mon.noname-a con 0x7ff3b4143d10 addr v1:172.21.15.84:6789/0 2024-03-29T11:29:37.758 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:37.765+0000 7ff3b8f23640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3b406ef80 con 0x7ff3b4143d10 2024-03-29T11:29:37.759 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:37.765+0000 7ff3b8f23640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3b406f260 con 0x7ff3b40b1f60 2024-03-29T11:29:37.759 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:37.765+0000 7ff3b8f23640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3b40b40c0 con 0x7ff3b4071eb0 2024-03-29T11:29:37.759 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:37.765+0000 7ff3b8f23640 10 monclient(hunting): _renew_subs 2024-03-29T11:29:37.759 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:37.765+0000 7ff3b8f23640 10 monclient(hunting): authenticate will time out at 1439.059592s 2024-03-29T11:29:37.759 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:37.765+0000 7ff3b1d9b640 1 --1- >> v1:172.21.15.84:6790/0 conn(0x7ff3b4071eb0 0x7ff3b4146470 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.84:6790/0 says I am v1:172.21.15.84:50780/0 (socket says 172.21.15.84:50780) 2024-03-29T11:29:37.759 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:37.765+0000 7ff3b1d9b640 1 -- 172.21.15.84:0/1952433525 learned_addr learned my addr 172.21.15.84:0/1952433525 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-29T11:29:37.759 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:37.766+0000 7ff3b159a640 1 -- 172.21.15.84:0/1952433525 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2398942430 0 0) 0x7ff3b40b40c0 con 0x7ff3b4071eb0 2024-03-29T11:29:37.759 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:37.766+0000 7ff3b159a640 10 monclient(hunting): _negotiate 2024-03-29T11:29:37.759 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:37.766+0000 7ff3b159a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:29:37.759 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:37.766+0000 7ff3b159a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:29:37.759 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:37.766+0000 7ff3b159a640 10 monclient(hunting): my global_id is 4649 2024-03-29T11:29:37.759 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:37.766+0000 7ff3b159a640 1 -- 172.21.15.84:0/1952433525 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff39c003520 con 0x7ff3b4071eb0 2024-03-29T11:29:37.761 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:37.767+0000 7ff3b159a640 1 -- 172.21.15.84:0/1952433525 <== mon.2 v1:172.21.15.84:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3258811333 0 0) 0x7ff39c003520 con 0x7ff3b4071eb0 2024-03-29T11:29:37.761 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:37.767+0000 7ff3b159a640 1 -- 172.21.15.84:0/1952433525 >> v1:172.21.15.179:6789/0 conn(0x7ff3b40b1f60 legacy=0x7ff3b4143600 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:29:37.761 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:37.769+0000 7ff3b159a640 1 -- 172.21.15.84:0/1952433525 >> v1:172.21.15.84:6789/0 conn(0x7ff3b4143d10 legacy=0x7ff3b4146b80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:29:37.763 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:37.771+0000 7ff3b159a640 10 monclient: _finish_hunting 0 2024-03-29T11:29:37.763 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:37.771+0000 7ff3b159a640 1 monclient: found mon.noname-b 2024-03-29T11:29:37.763 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:37.771+0000 7ff3b159a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:29:37.763 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:37.771+0000 7ff3b159a640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.84:6790/0 2024-03-29T11:29:37.763 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:37.771+0000 7ff3b159a640 1 -- 172.21.15.84:0/1952433525 --> v1:172.21.15.84:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3b4155d70 con 0x7ff3b4071eb0 2024-03-29T11:29:37.763 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:37.771+0000 7ff3b159a640 10 monclient: _finish_auth 0 2024-03-29T11:29:37.763 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:37.771+0000 7ff3b159a640 10 monclient: _check_auth_tickets 2024-03-29T11:29:37.763 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:37.771+0000 7ff3b159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:29:37.763 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:37.771+0000 7ff3b159a640 1 -- 172.21.15.84:0/1952433525 <== mon.2 v1:172.21.15.84:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4236868694 0 0) 0x7ff3a8001690 con 0x7ff3b4071eb0 2024-03-29T11:29:37.765 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:37.771+0000 7ff3b159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:29:37.765 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:37.771+0000 7ff3b159a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:29:37.765 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:37.771+0000 7ff3b159a640 10 monclient: dump: 2024-03-29T11:29:37.765 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T11:29:37.765 INFO:teuthology.misc.health.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:29:37.765 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T11:18:53.864025+0000 2024-03-29T11:29:37.765 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T11:18:53.864025+0000 2024-03-29T11:29:37.765 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T11:29:37.765 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:29:37.765 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T11:29:37.765 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T11:29:37.766 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T11:29:37.766 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:29:37.766 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:37.771+0000 7ff3b159a640 1 -- 172.21.15.84:0/1952433525 <== mon.2 v1:172.21.15.84:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff3a8001860 con 0x7ff3b4071eb0 2024-03-29T11:29:37.766 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:37.771+0000 7ff3b159a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:29:37.766 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:37.771+0000 7ff3b159a640 1 -- 172.21.15.84:0/1952433525 <== mon.2 v1:172.21.15.84:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4236868694 0 0) 0x7ff3a8001cb0 con 0x7ff3b4071eb0 2024-03-29T11:29:37.766 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:37.771+0000 7ff3b159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:29:37.766 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:37.771+0000 7ff3b159a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:29:37.766 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:37.771+0000 7ff3b159a640 10 monclient: dump: 2024-03-29T11:29:37.766 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T11:29:37.767 INFO:teuthology.misc.health.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:29:37.767 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T11:18:53.864025+0000 2024-03-29T11:29:37.767 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T11:18:53.864025+0000 2024-03-29T11:29:37.767 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T11:29:37.767 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:29:37.767 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T11:29:37.767 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T11:29:37.767 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T11:29:37.767 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:29:37.767 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:37.772+0000 7ff3b8f23640 5 monclient: authenticate success, global_id 4649 2024-03-29T11:29:37.768 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:37.772+0000 7ff3b8f23640 10 monclient: get_monmap_and_config success 2024-03-29T11:29:37.768 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:37.772+0000 7ff3b8f23640 4 set_mon_vals no callback set 2024-03-29T11:29:37.768 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:37.773+0000 7ff3b8f23640 10 monclient: shutdown 2024-03-29T11:29:37.768 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:37.773+0000 7ff3b8f23640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:29:37.768 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:37.773+0000 7ff3b8f23640 1 -- 172.21.15.84:0/1952433525 >> v1:172.21.15.84:6790/0 conn(0x7ff3b4071eb0 legacy=0x7ff3b4146470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:37.769 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:37.775+0000 7ff3b8f23640 1 -- 172.21.15.84:0/1952433525 shutdown_connections 2024-03-29T11:29:37.769 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:37.775+0000 7ff3b8f23640 1 -- 172.21.15.84:0/1952433525 >> 172.21.15.84:0/1952433525 conn(0x7ff3b4066a00 msgr2=0x7ff3b4066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:29:37.769 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:37.775+0000 7ff3b8f23640 1 -- 172.21.15.84:0/1952433525 shutdown_connections 2024-03-29T11:29:37.769 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:37.776+0000 7ff3b8f23640 1 -- 172.21.15.84:0/1952433525 wait complete. 2024-03-29T11:29:37.770 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:37.778+0000 7ff3b8f23640 10 monclient: build_initial_monmap 2024-03-29T11:29:37.772 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:37.778+0000 7ff3b8f23640 10 monclient: monmap: 2024-03-29T11:29:37.772 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-29T11:29:37.773 INFO:teuthology.misc.health.smithi084.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:29:37.773 INFO:teuthology.misc.health.smithi084.stderr:last_changed 0.000000 2024-03-29T11:29:37.773 INFO:teuthology.misc.health.smithi084.stderr:created 0.000000 2024-03-29T11:29:37.773 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-29T11:29:37.773 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:29:37.773 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-29T11:29:37.773 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.noname-b 2024-03-29T11:29:37.773 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-c 2024-03-29T11:29:37.773 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:29:37.773 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:37.778+0000 7ff3b8f23640 1 librados: starting msgr at 2024-03-29T11:29:37.773 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:37.778+0000 7ff3b8f23640 1 librados: starting objecter 2024-03-29T11:29:37.773 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:37.779+0000 7ff3b8f23640 1 Processor -- start 2024-03-29T11:29:37.774 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:37.779+0000 7ff3b8f23640 1 -- start start 2024-03-29T11:29:37.774 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:37.779+0000 7ff3b8f23640 1 librados: setting wanted keys 2024-03-29T11:29:37.774 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:37.779+0000 7ff3b8f23640 1 librados: calling monclient init 2024-03-29T11:29:37.774 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:37.779+0000 7ff3b8f23640 10 monclient: init 2024-03-29T11:29:37.774 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:37.779+0000 7ff3b8f23640 10 monclient: _reopen_session rank -1 2024-03-29T11:29:37.774 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:37.780+0000 7ff3b8f23640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:29:37.775 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:37.780+0000 7ff3b8f23640 10 monclient(hunting): picked mon.noname-a con 0x7ff3b40b1f60 addr v1:172.21.15.84:6789/0 2024-03-29T11:29:37.775 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:37.780+0000 7ff3b8f23640 10 monclient(hunting): picked mon.noname-b con 0x7ff3b4143d10 addr v1:172.21.15.179:6789/0 2024-03-29T11:29:37.775 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:37.780+0000 7ff3b8f23640 10 monclient(hunting): picked mon.noname-c con 0x7ff3b4142b30 addr v1:172.21.15.84:6790/0 2024-03-29T11:29:37.775 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:37.780+0000 7ff3b8f23640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3b40702d0 con 0x7ff3b40b1f60 2024-03-29T11:29:37.775 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:37.780+0000 7ff3b8f23640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3b4133aa0 con 0x7ff3b4143d10 2024-03-29T11:29:37.775 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:37.780+0000 7ff3b8f23640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3b4071390 con 0x7ff3b4142b30 2024-03-29T11:29:37.775 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:37.780+0000 7ff3b8f23640 10 monclient(hunting): _renew_subs 2024-03-29T11:29:37.775 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:37.780+0000 7ff3b8f23640 10 monclient(hunting): authenticate will time out at 1439.074190s 2024-03-29T11:29:37.781 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:37.783+0000 7ff3b259c640 1 --1- >> v1:172.21.15.84:6789/0 conn(0x7ff3b40b1f60 0x7ff3b406f7f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.84:6789/0 says I am v1:172.21.15.84:54567/0 (socket says 172.21.15.84:54567) 2024-03-29T11:29:37.782 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:37.783+0000 7ff3b2d9d640 1 --1- >> v1:172.21.15.84:6790/0 conn(0x7ff3b4142b30 0x7ff3b4133390 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.84:6790/0 says I am v1:172.21.15.84:52008/0 (socket says 172.21.15.84:52008) 2024-03-29T11:29:37.782 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:37.783+0000 7ff3b2d9d640 1 -- 172.21.15.84:0/404537664 learned_addr learned my addr 172.21.15.84:0/404537664 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-29T11:29:37.782 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:37.783+0000 7ff3a2ffd640 10 client.?.objecter ms_handle_connect 0x7ff3b4142b30 2024-03-29T11:29:37.782 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:37.783+0000 7ff3a2ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:29:37.782 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:37.783+0000 7ff3a2ffd640 10 client.?.objecter ms_handle_connect 0x7ff3b40b1f60 2024-03-29T11:29:37.782 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:37.783+0000 7ff3a2ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:29:37.782 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:37.784+0000 7ff3a2ffd640 1 -- 172.21.15.84:0/404537664 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3953778832 0 0) 0x7ff3b4071390 con 0x7ff3b4142b30 2024-03-29T11:29:37.782 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:37.784+0000 7ff3a2ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:29:37.782 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:37.784+0000 7ff3a2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:29:37.782 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:37.784+0000 7ff3a2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:29:37.782 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:37.784+0000 7ff3a2ffd640 10 monclient(hunting): my global_id is 4652 2024-03-29T11:29:37.782 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:37.784+0000 7ff3a2ffd640 1 -- 172.21.15.84:0/404537664 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff394004eb0 con 0x7ff3b4142b30 2024-03-29T11:29:37.782 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:37.784+0000 7ff3a2ffd640 1 -- 172.21.15.84:0/404537664 <== mon.0 v1:172.21.15.84:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3325268882 0 0) 0x7ff3b40702d0 con 0x7ff3b40b1f60 2024-03-29T11:29:37.783 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:37.784+0000 7ff3a2ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:29:37.783 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:37.784+0000 7ff3a2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:29:37.783 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:37.784+0000 7ff3a2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:29:37.783 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:37.784+0000 7ff3a2ffd640 10 monclient(hunting): my global_id is 4578 2024-03-29T11:29:37.783 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:37.784+0000 7ff3a2ffd640 1 -- 172.21.15.84:0/404537664 --> v1:172.21.15.84:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff3b4071390 con 0x7ff3b40b1f60 2024-03-29T11:29:37.783 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:37.784+0000 7ff3a2ffd640 1 -- 172.21.15.84:0/404537664 <== mon.2 v1:172.21.15.84:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2942215653 0 0) 0x7ff394004eb0 con 0x7ff3b4142b30 2024-03-29T11:29:37.783 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:37.785+0000 7ff3a2ffd640 1 -- 172.21.15.84:0/404537664 --> v1:172.21.15.84:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff3b40702d0 con 0x7ff3b4142b30 2024-03-29T11:29:37.783 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:37.785+0000 7ff3a2ffd640 1 -- 172.21.15.84:0/404537664 <== mon.2 v1:172.21.15.84:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4236868694 0 0) 0x7ff3a4003f60 con 0x7ff3b4142b30 2024-03-29T11:29:37.783 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:37.785+0000 7ff3a2ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:29:37.783 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:37.785+0000 7ff3a2ffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:29:37.783 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:37.785+0000 7ff3a2ffd640 10 monclient(hunting): dump: 2024-03-29T11:29:37.783 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T11:29:37.783 INFO:teuthology.misc.health.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:29:37.783 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T11:18:53.864025+0000 2024-03-29T11:29:37.784 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T11:18:53.864025+0000 2024-03-29T11:29:37.784 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T11:29:37.784 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:29:37.784 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T11:29:37.784 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T11:29:37.784 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T11:29:37.784 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:29:37.784 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:37.785+0000 7ff3a2ffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:29:37.784 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:37.785+0000 7ff3a2ffd640 1 -- 172.21.15.84:0/404537664 <== mon.0 v1:172.21.15.84:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3862921357 0 0) 0x7ff3b4071390 con 0x7ff3b40b1f60 2024-03-29T11:29:37.784 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:37.785+0000 7ff3a2ffd640 1 -- 172.21.15.84:0/404537664 --> v1:172.21.15.84:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff394004eb0 con 0x7ff3b40b1f60 2024-03-29T11:29:37.784 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:37.785+0000 7ff3a2ffd640 1 -- 172.21.15.84:0/404537664 <== mon.2 v1:172.21.15.84:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 161773321 0 0) 0x7ff3b40702d0 con 0x7ff3b4142b30 2024-03-29T11:29:37.784 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:37.785+0000 7ff3a2ffd640 1 -- 172.21.15.84:0/404537664 >> v1:172.21.15.179:6789/0 conn(0x7ff3b4143d10 legacy=0x7ff3b4132c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:37.784 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:37.785+0000 7ff3a2ffd640 1 -- 172.21.15.84:0/404537664 >> v1:172.21.15.84:6789/0 conn(0x7ff3b40b1f60 legacy=0x7ff3b406f7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:37.784 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:37.786+0000 7ff3a2ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:29:37.785 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:37.786+0000 7ff3a2ffd640 1 monclient: found mon.c 2024-03-29T11:29:37.785 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:37.786+0000 7ff3a2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:29:37.785 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:37.786+0000 7ff3a2ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-29T11:29:37.785 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:37.786+0000 7ff3a2ffd640 1 -- 172.21.15.84:0/404537664 --> v1:172.21.15.84:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3b4139270 con 0x7ff3b4142b30 2024-03-29T11:29:37.785 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:37.786+0000 7ff3a2ffd640 10 monclient: _finish_auth 0 2024-03-29T11:29:37.785 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:37.786+0000 7ff3a2ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:29:37.785 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:37.786+0000 7ff3a2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:29:37.785 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:37.786+0000 7ff3b8f23640 5 monclient: authenticate success, global_id 4652 2024-03-29T11:29:37.785 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:37.786+0000 7ff3b8f23640 10 monclient: _renew_subs 2024-03-29T11:29:37.785 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:37.786+0000 7ff3b8f23640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-29T11:29:37.785 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:37.786+0000 7ff3b8f23640 1 -- 172.21.15.84:0/404537664 --> v1:172.21.15.84:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff3b4139550 con 0x7ff3b4142b30 2024-03-29T11:29:37.785 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:37.786+0000 7ff3b8f23640 10 client.4652.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:29:37.785 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:37.786+0000 7ff3b8f23640 10 monclient: _renew_subs 2024-03-29T11:29:37.785 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:37.786+0000 7ff3b8f23640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-29T11:29:37.785 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:37.786+0000 7ff3b8f23640 1 -- 172.21.15.84:0/404537664 --> v1:172.21.15.84:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff3b4070ed0 con 0x7ff3b4142b30 2024-03-29T11:29:37.786 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:37.786+0000 7ff3b8f23640 1 librados: init done 2024-03-29T11:29:37.786 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:37.788+0000 7ff3a2ffd640 1 -- 172.21.15.84:0/404537664 <== mon.2 v1:172.21.15.84:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff3a40040c0 con 0x7ff3b4142b30 2024-03-29T11:29:37.786 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:37.788+0000 7ff3b8f23640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:29:37.786 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:37.788+0000 7ff3b8f23640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:29:37.786 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:37.788+0000 7ff3b8f23640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-29T11:29:37.786 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:37.788+0000 7ff3b8f23640 1 -- 172.21.15.84:0/404537664 --> v1:172.21.15.84:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff3b4075f10 con 0x7ff3b4142b30 2024-03-29T11:29:37.786 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:37.788+0000 7ff3a2ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:29:37.786 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:37.788+0000 7ff3b0d99640 4 set_mon_vals no callback set 2024-03-29T11:29:37.786 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:37.789+0000 7ff3a2ffd640 1 -- 172.21.15.84:0/404537664 <== mon.2 v1:172.21.15.84:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4236868694 0 0) 0x7ff3a4001070 con 0x7ff3b4142b30 2024-03-29T11:29:37.786 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:37.789+0000 7ff3a2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:29:37.786 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:37.789+0000 7ff3a2ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:29:37.786 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:37.789+0000 7ff3a2ffd640 10 monclient: dump: 2024-03-29T11:29:37.786 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T11:29:37.786 INFO:teuthology.misc.health.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:29:37.787 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T11:18:53.864025+0000 2024-03-29T11:29:37.787 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T11:18:53.864025+0000 2024-03-29T11:29:37.787 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T11:29:37.787 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:29:37.787 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T11:29:37.787 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T11:29:37.787 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T11:29:37.787 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:29:37.788 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:37.790+0000 7ff3a2ffd640 1 -- 172.21.15.84:0/404537664 <== mon.2 v1:172.21.15.84:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4033942909 0 0) 0x7ff3a4001250 con 0x7ff3b4142b30 2024-03-29T11:29:37.788 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:37.791+0000 7ff3a2ffd640 1 -- 172.21.15.84:0/404537664 <== mon.2 v1:172.21.15.84:6790/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 156031568 0 0) 0x7ff3a40848f0 con 0x7ff3b4142b30 2024-03-29T11:29:37.788 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:37.791+0000 7ff3a2ffd640 10 client.4652.objecter ms_dispatch 0x7ff3b4000b90 osd_map(11..11 src has 1..11) v4 2024-03-29T11:29:37.788 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:37.791+0000 7ff3a2ffd640 3 client.4652.objecter handle_osd_map got epochs [11,11] > 0 2024-03-29T11:29:37.788 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:37.791+0000 7ff3a2ffd640 3 client.4652.objecter handle_osd_map decoding full epoch 11 2024-03-29T11:29:37.788 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:37.792+0000 7ff3a2ffd640 20 client.4652.objecter dump_active .. 0 homeless 2024-03-29T11:29:37.789 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:37.792+0000 7ff3b259c640 10 monclient: get_auth_request con 0x7ff39406dea0 auth_method 0 2024-03-29T11:29:37.789 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:37.792+0000 7ff3a2ffd640 10 client.4652.objecter ms_handle_connect 0x7ff39406dea0 2024-03-29T11:29:37.789 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:37.798+0000 7ff3a2ffd640 1 -- 172.21.15.84:0/404537664 <== mon.2 v1:172.21.15.84:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7ff3a40501d0 con 0x7ff3b4142b30 2024-03-29T11:29:37.790 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:37.798+0000 7ff3a2ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:29:37.790 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:37.798+0000 7ff3a2ffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:29:38.109 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:38.117+0000 7ff3b8f23640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:29:38.109 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:38.117+0000 7ff3b8f23640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:29:38.109 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:38.117+0000 7ff3b8f23640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-29T11:29:38.109 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:38.117+0000 7ff3b8f23640 1 -- 172.21.15.84:0/404537664 --> v1:172.21.15.84:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff3b406ff00 con 0x7ff3b4142b30 2024-03-29T11:29:38.110 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:38.118+0000 7ff3a2ffd640 1 -- 172.21.15.84:0/404537664 <== mon.2 v1:172.21.15.84:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 782149409) 0x7ff3a4052100 con 0x7ff3b4142b30 2024-03-29T11:29:38.110 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:38.118+0000 7ff3a2ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:29:38.110 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:38.118+0000 7ff3a2ffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:29:38.111 INFO:teuthology.misc.health.smithi084.stdout: 2024-03-29T11:29:38.111 INFO:teuthology.misc.health.smithi084.stdout:{"epoch":11,"fsid":"2c3e002f-9eb0-4860-853b-4ef1ef562213","created":"2024-03-29T11:19:32.029169+0000","modified":"2024-03-29T11:29:36.955202+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:29:36.955202+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"5c39e276-2034-4716-bbc0-57a634eefa92","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"cd580e99-109f-4bf1-88c0-a5987bc8b9ed","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"11e60a5a-051d-453f-a190-66387cb35592","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"2c329fbe-bbc4-4030-bf87-d711cad04679","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f4213d4a-bd36-4433-91ae-d7e2b02481d3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"f0cf44ea-6055-48af-805b-aed858102ff9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"d895eaa0-cc0b-4ee4-9b5a-2dfcd89a60b7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"9bc5a7bc-8f52-4db9-96d6-de5e68d52eaa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:29:38.114 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:38.122+0000 7ff3b8f23640 10 librados: watch_flush enter 2024-03-29T11:29:38.114 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:38.122+0000 7ff3b8f23640 10 librados: watch_flush exit 2024-03-29T11:29:38.114 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:38.122+0000 7ff3b8f23640 20 client.4652.objecter shutdown clearing up homeless session... 2024-03-29T11:29:38.114 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:38.122+0000 7ff3b8f23640 10 client.4652.objecter successfully canceled tick 2024-03-29T11:29:38.115 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:38.123+0000 7ff3b8f23640 1 -- 172.21.15.84:0/404537664 >> v1:172.21.15.179:6816/2981134980 conn(0x7ff39406dea0 legacy=0x7ff394070360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:38.115 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:38.123+0000 7ff3b8f23640 10 monclient: shutdown 2024-03-29T11:29:38.115 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:38.123+0000 7ff3b8f23640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:29:38.115 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:38.123+0000 7ff3b8f23640 1 -- 172.21.15.84:0/404537664 >> v1:172.21.15.84:6790/0 conn(0x7ff3b4142b30 legacy=0x7ff3b4133390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:38.118 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:38.126+0000 7ff3b8f23640 1 -- 172.21.15.84:0/404537664 shutdown_connections 2024-03-29T11:29:38.118 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:38.126+0000 7ff3b8f23640 1 -- 172.21.15.84:0/404537664 >> 172.21.15.84:0/404537664 conn(0x7ff3b4066a00 msgr2=0x7ff3b40b5810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:29:38.118 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:38.127+0000 7ff3b8f23640 1 -- 172.21.15.84:0/404537664 shutdown_connections 2024-03-29T11:29:38.118 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:38.127+0000 7ff3b8f23640 1 -- 172.21.15.84:0/404537664 wait complete. 2024-03-29T11:29:38.118 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:38.127+0000 7ff3b8f23640 1 librados: shutdown 2024-03-29T11:29:38.134 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:29:44.942 DEBUG:teuthology.orchestra.run.smithi084:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:29:45.089 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:45.097+0000 7f8b5a4c0640 10 monclient: get_monmap_and_config 2024-03-29T11:29:45.089 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:45.097+0000 7f8b5a4c0640 10 monclient: build_initial_monmap 2024-03-29T11:29:45.089 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:45.097+0000 7f8b5a4c0640 10 monclient: monmap: 2024-03-29T11:29:45.089 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-29T11:29:45.089 INFO:teuthology.misc.health.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:29:45.089 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T11:29:45.098308+0000 2024-03-29T11:29:45.089 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T11:29:45.098308+0000 2024-03-29T11:29:45.089 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-29T11:29:45.089 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:29:45.089 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-29T11:29:45.090 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.noname-c 2024-03-29T11:29:45.090 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-b 2024-03-29T11:29:45.090 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:29:45.091 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:45.098+0000 7f8b5a4c0640 1 Processor -- start 2024-03-29T11:29:45.091 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:45.098+0000 7f8b5a4c0640 1 -- start start 2024-03-29T11:29:45.091 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:45.098+0000 7f8b5a4c0640 10 monclient: init 2024-03-29T11:29:45.091 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:45.099+0000 7f8b5a4c0640 10 monclient: _reopen_session rank -1 2024-03-29T11:29:45.091 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:45.099+0000 7f8b5a4c0640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:29:45.091 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:45.099+0000 7f8b5a4c0640 10 monclient(hunting): picked mon.noname-a con 0x7f8b541422b0 addr v1:172.21.15.84:6789/0 2024-03-29T11:29:45.091 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:45.099+0000 7f8b5a4c0640 10 monclient(hunting): picked mon.noname-b con 0x7f8b54142690 addr v1:172.21.15.84:6790/0 2024-03-29T11:29:45.091 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:45.099+0000 7f8b5a4c0640 10 monclient(hunting): picked mon.noname-c con 0x7f8b540b17d0 addr v1:172.21.15.179:6789/0 2024-03-29T11:29:45.091 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:45.099+0000 7f8b5a4c0640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8b54142c80 con 0x7f8b541422b0 2024-03-29T11:29:45.091 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:45.099+0000 7f8b5a4c0640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8b540aae40 con 0x7f8b540b17d0 2024-03-29T11:29:45.091 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:45.099+0000 7f8b5a4c0640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8b5413fbd0 con 0x7f8b54142690 2024-03-29T11:29:45.091 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:45.099+0000 7f8b5a4c0640 10 monclient(hunting): _renew_subs 2024-03-29T11:29:45.091 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:45.099+0000 7f8b5a4c0640 10 monclient(hunting): authenticate will time out at 1446.393420s 2024-03-29T11:29:45.094 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:45.102+0000 7f8b58a5d640 1 --1- >> v1:172.21.15.179:6789/0 conn(0x7f8b540b17d0 0x7f8b540b2ca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.179:6789/0 says I am v1:172.21.15.84:59461/0 (socket says 172.21.15.84:59461) 2024-03-29T11:29:45.094 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:45.102+0000 7f8b58a5d640 1 -- 172.21.15.84:0/878697062 learned_addr learned my addr 172.21.15.84:0/878697062 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-29T11:29:45.094 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:45.103+0000 7f8b52ffd640 1 -- 172.21.15.84:0/878697062 <== mon.1 v1:172.21.15.179:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 846778781 0 0) 0x7f8b540aae40 con 0x7f8b540b17d0 2024-03-29T11:29:45.095 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:45.103+0000 7f8b52ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:29:45.095 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:45.104+0000 7f8b52ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:29:45.097 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:45.104+0000 7f8b52ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:29:45.098 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:45.104+0000 7f8b52ffd640 10 monclient(hunting): my global_id is 4531 2024-03-29T11:29:45.098 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:45.104+0000 7f8b52ffd640 1 -- 172.21.15.84:0/878697062 --> v1:172.21.15.179:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8b3c003520 con 0x7f8b540b17d0 2024-03-29T11:29:45.098 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:45.104+0000 7f8b52ffd640 1 -- 172.21.15.84:0/878697062 <== mon.0 v1:172.21.15.84:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 517693353 0 0) 0x7f8b54142c80 con 0x7f8b541422b0 2024-03-29T11:29:45.098 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:45.104+0000 7f8b52ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:29:45.098 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:45.104+0000 7f8b52ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:29:45.098 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:45.104+0000 7f8b52ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:29:45.098 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:45.104+0000 7f8b52ffd640 10 monclient(hunting): my global_id is 4581 2024-03-29T11:29:45.098 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:45.104+0000 7f8b52ffd640 1 -- 172.21.15.84:0/878697062 --> v1:172.21.15.84:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8b540aae40 con 0x7f8b541422b0 2024-03-29T11:29:45.098 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:45.104+0000 7f8b52ffd640 1 -- 172.21.15.84:0/878697062 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 63729211 0 0) 0x7f8b5413fbd0 con 0x7f8b54142690 2024-03-29T11:29:45.098 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:45.104+0000 7f8b52ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:29:45.098 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:45.104+0000 7f8b52ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:29:45.098 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:45.104+0000 7f8b52ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:29:45.098 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:45.104+0000 7f8b52ffd640 10 monclient(hunting): my global_id is 4655 2024-03-29T11:29:45.099 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:45.104+0000 7f8b52ffd640 1 -- 172.21.15.84:0/878697062 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8b54142c80 con 0x7f8b54142690 2024-03-29T11:29:45.099 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:45.104+0000 7f8b52ffd640 1 -- 172.21.15.84:0/878697062 <== mon.1 v1:172.21.15.179:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4239122070 0 0) 0x7f8b3c003520 con 0x7f8b540b17d0 2024-03-29T11:29:45.099 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:45.104+0000 7f8b52ffd640 1 -- 172.21.15.84:0/878697062 >> v1:172.21.15.84:6790/0 conn(0x7f8b54142690 legacy=0x7f8b540b2590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:45.099 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:45.104+0000 7f8b52ffd640 1 -- 172.21.15.84:0/878697062 >> v1:172.21.15.84:6789/0 conn(0x7f8b541422b0 legacy=0x7f8b5413f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:45.099 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:45.104+0000 7f8b52ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:29:45.099 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:45.104+0000 7f8b52ffd640 1 monclient: found mon.noname-c 2024-03-29T11:29:45.099 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:45.104+0000 7f8b52ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:29:45.099 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:45.104+0000 7f8b52ffd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.179:6789/0 2024-03-29T11:29:45.099 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:45.104+0000 7f8b52ffd640 1 -- 172.21.15.84:0/878697062 --> v1:172.21.15.179:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b54151940 con 0x7f8b540b17d0 2024-03-29T11:29:45.099 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:45.104+0000 7f8b52ffd640 10 monclient: _finish_auth 0 2024-03-29T11:29:45.099 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:45.104+0000 7f8b52ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:29:45.099 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:45.104+0000 7f8b52ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:29:45.099 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:45.104+0000 7f8b52ffd640 1 -- 172.21.15.84:0/878697062 <== mon.1 v1:172.21.15.179:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4236868694 0 0) 0x7f8b4c001a10 con 0x7f8b540b17d0 2024-03-29T11:29:45.099 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:45.104+0000 7f8b52ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:29:45.100 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:45.104+0000 7f8b52ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:29:45.100 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:45.104+0000 7f8b52ffd640 10 monclient: dump: 2024-03-29T11:29:45.100 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T11:29:45.100 INFO:teuthology.misc.health.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:29:45.100 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T11:18:53.864025+0000 2024-03-29T11:29:45.100 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T11:18:53.864025+0000 2024-03-29T11:29:45.100 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T11:29:45.100 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:29:45.100 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T11:29:45.100 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T11:29:45.100 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T11:29:45.100 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:29:45.100 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:45.105+0000 7f8b52ffd640 1 -- 172.21.15.84:0/878697062 <== mon.1 v1:172.21.15.179:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8b4c001bf0 con 0x7f8b540b17d0 2024-03-29T11:29:45.100 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:45.105+0000 7f8b52ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:29:45.100 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:45.105+0000 7f8b52ffd640 1 -- 172.21.15.84:0/878697062 <== mon.1 v1:172.21.15.179:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4236868694 0 0) 0x7f8b4c001f80 con 0x7f8b540b17d0 2024-03-29T11:29:45.101 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:45.105+0000 7f8b52ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:29:45.101 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:45.105+0000 7f8b52ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:29:45.101 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:45.105+0000 7f8b52ffd640 10 monclient: dump: 2024-03-29T11:29:45.101 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T11:29:45.101 INFO:teuthology.misc.health.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:29:45.101 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T11:18:53.864025+0000 2024-03-29T11:29:45.101 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T11:18:53.864025+0000 2024-03-29T11:29:45.101 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T11:29:45.101 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:29:45.101 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T11:29:45.101 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T11:29:45.101 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T11:29:45.101 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:29:45.101 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:45.105+0000 7f8b5a4c0640 5 monclient: authenticate success, global_id 4531 2024-03-29T11:29:45.101 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:45.105+0000 7f8b5a4c0640 10 monclient: get_monmap_and_config success 2024-03-29T11:29:45.102 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:45.105+0000 7f8b5a4c0640 4 set_mon_vals no callback set 2024-03-29T11:29:45.102 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:45.106+0000 7f8b5a4c0640 10 monclient: shutdown 2024-03-29T11:29:45.102 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:45.106+0000 7f8b5a4c0640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:29:45.102 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:45.106+0000 7f8b5a4c0640 1 -- 172.21.15.84:0/878697062 >> v1:172.21.15.179:6789/0 conn(0x7f8b540b17d0 legacy=0x7f8b540b2ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:45.102 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:45.106+0000 7f8b5a4c0640 1 -- 172.21.15.84:0/878697062 shutdown_connections 2024-03-29T11:29:45.102 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:45.106+0000 7f8b5a4c0640 1 -- 172.21.15.84:0/878697062 >> 172.21.15.84:0/878697062 conn(0x7f8b54066a00 msgr2=0x7f8b54066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:29:45.108 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:45.106+0000 7f8b5a4c0640 1 -- 172.21.15.84:0/878697062 shutdown_connections 2024-03-29T11:29:45.108 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:45.106+0000 7f8b5a4c0640 1 -- 172.21.15.84:0/878697062 wait complete. 2024-03-29T11:29:45.108 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:45.107+0000 7f8b5a4c0640 10 monclient: build_initial_monmap 2024-03-29T11:29:45.108 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:45.107+0000 7f8b5a4c0640 10 monclient: monmap: 2024-03-29T11:29:45.108 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-29T11:29:45.108 INFO:teuthology.misc.health.smithi084.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:29:45.108 INFO:teuthology.misc.health.smithi084.stderr:last_changed 0.000000 2024-03-29T11:29:45.109 INFO:teuthology.misc.health.smithi084.stderr:created 0.000000 2024-03-29T11:29:45.109 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-29T11:29:45.109 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:29:45.109 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-29T11:29:45.109 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.noname-b 2024-03-29T11:29:45.109 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-c 2024-03-29T11:29:45.109 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:29:45.109 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:45.108+0000 7f8b5a4c0640 1 librados: starting msgr at 2024-03-29T11:29:45.109 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:45.108+0000 7f8b5a4c0640 1 librados: starting objecter 2024-03-29T11:29:45.109 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:45.108+0000 7f8b5a4c0640 1 Processor -- start 2024-03-29T11:29:45.109 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:45.108+0000 7f8b5a4c0640 1 -- start start 2024-03-29T11:29:45.109 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:45.108+0000 7f8b5a4c0640 1 librados: setting wanted keys 2024-03-29T11:29:45.109 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:45.108+0000 7f8b5a4c0640 1 librados: calling monclient init 2024-03-29T11:29:45.109 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:45.108+0000 7f8b5a4c0640 10 monclient: init 2024-03-29T11:29:45.110 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:45.109+0000 7f8b5a4c0640 10 monclient: _reopen_session rank -1 2024-03-29T11:29:45.110 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:45.109+0000 7f8b5a4c0640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:29:45.110 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:45.109+0000 7f8b5a4c0640 10 monclient(hunting): picked mon.noname-a con 0x7f8b540b17d0 addr v1:172.21.15.84:6789/0 2024-03-29T11:29:45.110 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:45.111+0000 7f8b5a4c0640 10 monclient(hunting): picked mon.noname-b con 0x7f8b541422b0 addr v1:172.21.15.179:6789/0 2024-03-29T11:29:45.110 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:45.111+0000 7f8b5a4c0640 10 monclient(hunting): picked mon.noname-c con 0x7f8b54142690 addr v1:172.21.15.84:6790/0 2024-03-29T11:29:45.110 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:45.111+0000 7f8b5a4c0640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8b540b56a0 con 0x7f8b540b17d0 2024-03-29T11:29:45.110 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:45.111+0000 7f8b5a4c0640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8b540abe80 con 0x7f8b541422b0 2024-03-29T11:29:45.110 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:45.111+0000 7f8b5a4c0640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8b540accb0 con 0x7f8b54142690 2024-03-29T11:29:45.110 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:45.111+0000 7f8b5a4c0640 10 monclient(hunting): _renew_subs 2024-03-29T11:29:45.110 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:45.111+0000 7f8b5a4c0640 10 monclient(hunting): authenticate will time out at 1446.405614s 2024-03-29T11:29:45.110 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:45.112+0000 7f8b58a5d640 1 --1- >> v1:172.21.15.84:6790/0 conn(0x7f8b54142690 0x7f8b541e2050 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.84:6790/0 says I am v1:172.21.15.84:36728/0 (socket says 172.21.15.84:36728) 2024-03-29T11:29:45.110 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:45.112+0000 7f8b58a5d640 1 -- 172.21.15.84:0/4056024878 learned_addr learned my addr 172.21.15.84:0/4056024878 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-29T11:29:45.110 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:45.112+0000 7f8b50ff9640 10 client.?.objecter ms_handle_connect 0x7f8b54142690 2024-03-29T11:29:45.110 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:45.112+0000 7f8b50ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:29:45.111 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:45.112+0000 7f8b50ff9640 10 client.?.objecter ms_handle_connect 0x7f8b541422b0 2024-03-29T11:29:45.111 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:45.112+0000 7f8b50ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:29:45.111 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:45.112+0000 7f8b50ff9640 1 -- 172.21.15.84:0/4056024878 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2151169058 0 0) 0x7f8b540accb0 con 0x7f8b54142690 2024-03-29T11:29:45.111 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:45.112+0000 7f8b50ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:29:45.111 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:45.112+0000 7f8b50ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:29:45.111 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:45.112+0000 7f8b50ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:29:45.111 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:45.112+0000 7f8b50ff9640 10 monclient(hunting): my global_id is 4658 2024-03-29T11:29:45.111 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:45.112+0000 7f8b50ff9640 1 -- 172.21.15.84:0/4056024878 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8b40004eb0 con 0x7f8b54142690 2024-03-29T11:29:45.111 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:45.112+0000 7f8b50ff9640 1 -- 172.21.15.84:0/4056024878 <== mon.1 v1:172.21.15.179:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4219959054 0 0) 0x7f8b540abe80 con 0x7f8b541422b0 2024-03-29T11:29:45.111 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:45.112+0000 7f8b50ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:29:45.111 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:45.112+0000 7f8b50ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:29:45.111 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:45.112+0000 7f8b50ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:29:45.111 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:45.112+0000 7f8b50ff9640 10 monclient(hunting): my global_id is 4534 2024-03-29T11:29:45.111 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:45.112+0000 7f8b50ff9640 1 -- 172.21.15.84:0/4056024878 --> v1:172.21.15.179:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8b540accb0 con 0x7f8b541422b0 2024-03-29T11:29:45.112 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:45.114+0000 7f8b50ff9640 1 -- 172.21.15.84:0/4056024878 <== mon.1 v1:172.21.15.179:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 577457215 0 0) 0x7f8b540accb0 con 0x7f8b541422b0 2024-03-29T11:29:45.112 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:45.114+0000 7f8b50ff9640 1 -- 172.21.15.84:0/4056024878 --> v1:172.21.15.179:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8b540abe80 con 0x7f8b541422b0 2024-03-29T11:29:45.112 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:45.114+0000 7f8b50ff9640 1 -- 172.21.15.84:0/4056024878 <== mon.1 v1:172.21.15.179:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4236868694 0 0) 0x7f8b44001740 con 0x7f8b541422b0 2024-03-29T11:29:45.112 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:45.114+0000 7f8b50ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:29:45.112 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:45.114+0000 7f8b50ff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:29:45.112 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:45.114+0000 7f8b50ff9640 10 monclient(hunting): dump: 2024-03-29T11:29:45.112 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T11:29:45.112 INFO:teuthology.misc.health.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:29:45.112 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T11:18:53.864025+0000 2024-03-29T11:29:45.112 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T11:18:53.864025+0000 2024-03-29T11:29:45.112 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T11:29:45.112 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:29:45.112 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T11:29:45.112 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T11:29:45.112 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T11:29:45.113 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:29:45.113 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:45.114+0000 7f8b50ff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:29:45.113 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:45.114+0000 7f8b50ff9640 10 client.?.objecter ms_handle_connect 0x7f8b540b17d0 2024-03-29T11:29:45.113 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:45.114+0000 7f8b50ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:29:45.113 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:45.114+0000 7f8b50ff9640 1 -- 172.21.15.84:0/4056024878 <== mon.2 v1:172.21.15.84:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3927471054 0 0) 0x7f8b40004eb0 con 0x7f8b54142690 2024-03-29T11:29:45.113 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:45.114+0000 7f8b50ff9640 1 -- 172.21.15.84:0/4056024878 --> v1:172.21.15.84:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8b540accb0 con 0x7f8b54142690 2024-03-29T11:29:45.113 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:45.114+0000 7f8b50ff9640 1 -- 172.21.15.84:0/4056024878 <== mon.2 v1:172.21.15.84:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4236868694 0 0) 0x7f8b4c001fc0 con 0x7f8b54142690 2024-03-29T11:29:45.113 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:45.114+0000 7f8b50ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:29:45.113 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:45.114+0000 7f8b50ff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:29:45.113 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:45.114+0000 7f8b50ff9640 10 monclient(hunting): dump: 2024-03-29T11:29:45.113 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T11:29:45.113 INFO:teuthology.misc.health.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:29:45.113 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T11:18:53.864025+0000 2024-03-29T11:29:45.113 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T11:18:53.864025+0000 2024-03-29T11:29:45.113 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T11:29:45.114 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:29:45.114 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T11:29:45.114 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T11:29:45.114 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T11:29:45.114 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:29:45.114 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:45.115+0000 7f8b50ff9640 1 -- 172.21.15.84:0/4056024878 <== mon.1 v1:172.21.15.179:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2471395145 0 0) 0x7f8b540abe80 con 0x7f8b541422b0 2024-03-29T11:29:45.114 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:45.115+0000 7f8b50ff9640 1 -- 172.21.15.84:0/4056024878 >> v1:172.21.15.84:6790/0 conn(0x7f8b54142690 legacy=0x7f8b541e2050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:45.114 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:45.115+0000 7f8b50ff9640 1 -- 172.21.15.84:0/4056024878 >> v1:172.21.15.84:6789/0 conn(0x7f8b540b17d0 legacy=0x7f8b540b80c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:45.114 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:45.115+0000 7f8b50ff9640 10 monclient: _finish_hunting 0 2024-03-29T11:29:45.114 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:45.115+0000 7f8b50ff9640 1 monclient: found mon.b 2024-03-29T11:29:45.114 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:45.115+0000 7f8b50ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:29:45.114 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:45.115+0000 7f8b50ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-29T11:29:45.114 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:45.115+0000 7f8b50ff9640 1 -- 172.21.15.84:0/4056024878 --> v1:172.21.15.179:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b540acf90 con 0x7f8b541422b0 2024-03-29T11:29:45.114 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:45.115+0000 7f8b50ff9640 10 monclient: _finish_auth 0 2024-03-29T11:29:45.115 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:45.115+0000 7f8b50ff9640 10 monclient: _check_auth_tickets 2024-03-29T11:29:45.115 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:45.115+0000 7f8b50ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:29:45.115 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:45.115+0000 7f8b5a4c0640 5 monclient: authenticate success, global_id 4534 2024-03-29T11:29:45.115 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:45.115+0000 7f8b5a4c0640 10 monclient: _renew_subs 2024-03-29T11:29:45.115 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:45.115+0000 7f8b5a4c0640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-29T11:29:45.115 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:45.115+0000 7f8b5a4c0640 1 -- 172.21.15.84:0/4056024878 --> v1:172.21.15.179:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8b540ab950 con 0x7f8b541422b0 2024-03-29T11:29:45.115 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:45.115+0000 7f8b5a4c0640 10 client.4534.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:29:45.115 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:45.115+0000 7f8b5a4c0640 10 monclient: _renew_subs 2024-03-29T11:29:45.115 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:45.115+0000 7f8b5a4c0640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-29T11:29:45.115 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:45.115+0000 7f8b5a4c0640 1 -- 172.21.15.84:0/4056024878 --> v1:172.21.15.179:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f8b540abc50 con 0x7f8b541422b0 2024-03-29T11:29:45.115 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:45.115+0000 7f8b5a4c0640 1 librados: init done 2024-03-29T11:29:45.117 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:45.117+0000 7f8b5a4c0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:29:45.118 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:45.117+0000 7f8b5a4c0640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:29:45.118 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:45.117+0000 7f8b5a4c0640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-29T11:29:45.118 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:45.117+0000 7f8b5a4c0640 1 -- 172.21.15.84:0/4056024878 --> v1:172.21.15.179:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8b54075f10 con 0x7f8b541422b0 2024-03-29T11:29:45.118 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:45.118+0000 7f8b50ff9640 1 -- 172.21.15.84:0/4056024878 <== mon.1 v1:172.21.15.179:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8b44001c50 con 0x7f8b541422b0 2024-03-29T11:29:45.118 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:45.118+0000 7f8b50ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:29:45.118 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:45.118+0000 7f8b50ff9640 1 -- 172.21.15.84:0/4056024878 <== mon.1 v1:172.21.15.179:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4236868694 0 0) 0x7f8b440037b0 con 0x7f8b541422b0 2024-03-29T11:29:45.118 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:45.118+0000 7f8b50ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:29:45.118 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:45.118+0000 7f8b50ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:29:45.118 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:45.118+0000 7f8b50ff9640 10 monclient: dump: 2024-03-29T11:29:45.118 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T11:29:45.118 INFO:teuthology.misc.health.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:29:45.118 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T11:18:53.864025+0000 2024-03-29T11:29:45.119 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T11:18:53.864025+0000 2024-03-29T11:29:45.119 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T11:29:45.119 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:29:45.119 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T11:29:45.119 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T11:29:45.119 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T11:29:45.119 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:29:45.119 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:45.119+0000 7f8b50ff9640 1 -- 172.21.15.84:0/4056024878 <== mon.1 v1:172.21.15.179:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4033942909 0 0) 0x7f8b44019780 con 0x7f8b541422b0 2024-03-29T11:29:45.119 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:45.118+0000 7f8b527fc640 4 set_mon_vals no callback set 2024-03-29T11:29:45.119 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:45.119+0000 7f8b50ff9640 1 -- 172.21.15.84:0/4056024878 <== mon.1 v1:172.21.15.179:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 156031568 0 0) 0x7f8b44083740 con 0x7f8b541422b0 2024-03-29T11:29:45.119 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:45.119+0000 7f8b50ff9640 10 client.4534.objecter ms_dispatch 0x7f8b54000b90 osd_map(11..11 src has 1..11) v4 2024-03-29T11:29:45.119 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:45.119+0000 7f8b50ff9640 3 client.4534.objecter handle_osd_map got epochs [11,11] > 0 2024-03-29T11:29:45.119 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:45.119+0000 7f8b50ff9640 3 client.4534.objecter handle_osd_map decoding full epoch 11 2024-03-29T11:29:45.119 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:45.119+0000 7f8b50ff9640 20 client.4534.objecter dump_active .. 0 homeless 2024-03-29T11:29:45.120 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:45.121+0000 7f8b53fff640 10 monclient: get_auth_request con 0x7f8b4006de50 auth_method 0 2024-03-29T11:29:45.120 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:45.122+0000 7f8b50ff9640 10 client.4534.objecter ms_handle_connect 0x7f8b4006de50 2024-03-29T11:29:45.120 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:45.125+0000 7f8b50ff9640 1 -- 172.21.15.84:0/4056024878 <== mon.1 v1:172.21.15.179:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f8b4404f0a0 con 0x7f8b541422b0 2024-03-29T11:29:45.120 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:45.125+0000 7f8b50ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:29:45.120 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:45.125+0000 7f8b50ff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:29:45.445 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:45.453+0000 7f8b5a4c0640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:29:45.445 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:45.453+0000 7f8b5a4c0640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:29:45.445 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:45.453+0000 7f8b5a4c0640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-29T11:29:45.445 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:45.453+0000 7f8b5a4c0640 1 -- 172.21.15.84:0/4056024878 --> v1:172.21.15.179:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8b540b14f0 con 0x7f8b541422b0 2024-03-29T11:29:45.446 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:45.454+0000 7f8b50ff9640 1 -- 172.21.15.84:0/4056024878 <== mon.1 v1:172.21.15.179:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 782149409) 0x7f8b44050fd0 con 0x7f8b541422b0 2024-03-29T11:29:45.446 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:45.454+0000 7f8b50ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:29:45.446 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:45.454+0000 7f8b50ff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:29:45.446 INFO:teuthology.misc.health.smithi084.stdout: 2024-03-29T11:29:45.446 INFO:teuthology.misc.health.smithi084.stdout:{"epoch":11,"fsid":"2c3e002f-9eb0-4860-853b-4ef1ef562213","created":"2024-03-29T11:19:32.029169+0000","modified":"2024-03-29T11:29:36.955202+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:29:36.955202+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"5c39e276-2034-4716-bbc0-57a634eefa92","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"cd580e99-109f-4bf1-88c0-a5987bc8b9ed","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"11e60a5a-051d-453f-a190-66387cb35592","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"2c329fbe-bbc4-4030-bf87-d711cad04679","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f4213d4a-bd36-4433-91ae-d7e2b02481d3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"f0cf44ea-6055-48af-805b-aed858102ff9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"d895eaa0-cc0b-4ee4-9b5a-2dfcd89a60b7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"9bc5a7bc-8f52-4db9-96d6-de5e68d52eaa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:29:45.449 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:45.457+0000 7f8b5a4c0640 10 librados: watch_flush enter 2024-03-29T11:29:45.449 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:45.458+0000 7f8b5a4c0640 10 librados: watch_flush exit 2024-03-29T11:29:45.449 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:45.458+0000 7f8b5a4c0640 20 client.4534.objecter shutdown clearing up homeless session... 2024-03-29T11:29:45.449 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:45.458+0000 7f8b5a4c0640 10 client.4534.objecter successfully canceled tick 2024-03-29T11:29:45.454 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:45.460+0000 7f8b5a4c0640 1 -- 172.21.15.84:0/4056024878 >> v1:172.21.15.179:6816/2981134980 conn(0x7f8b4006de50 legacy=0x7f8b40070310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:45.454 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:45.460+0000 7f8b5a4c0640 10 monclient: shutdown 2024-03-29T11:29:45.454 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:45.460+0000 7f8b5a4c0640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:29:45.454 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:45.460+0000 7f8b5a4c0640 1 -- 172.21.15.84:0/4056024878 >> v1:172.21.15.179:6789/0 conn(0x7f8b541422b0 legacy=0x7f8b541dae80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:45.454 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:45.461+0000 7f8b5a4c0640 1 -- 172.21.15.84:0/4056024878 shutdown_connections 2024-03-29T11:29:45.454 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:45.461+0000 7f8b5a4c0640 1 -- 172.21.15.84:0/4056024878 >> 172.21.15.84:0/4056024878 conn(0x7f8b54066a00 msgr2=0x7f8b54066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:29:45.454 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:45.462+0000 7f8b5a4c0640 1 -- 172.21.15.84:0/4056024878 shutdown_connections 2024-03-29T11:29:45.454 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:45.462+0000 7f8b5a4c0640 1 -- 172.21.15.84:0/4056024878 wait complete. 2024-03-29T11:29:45.455 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:45.462+0000 7f8b5a4c0640 1 librados: shutdown 2024-03-29T11:29:45.472 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:29:52.280 DEBUG:teuthology.orchestra.run.smithi084:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:29:52.428 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:52.436+0000 7f57225e8640 10 monclient: get_monmap_and_config 2024-03-29T11:29:52.428 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:52.436+0000 7f57225e8640 10 monclient: build_initial_monmap 2024-03-29T11:29:52.428 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:52.436+0000 7f57225e8640 10 monclient: monmap: 2024-03-29T11:29:52.428 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-29T11:29:52.428 INFO:teuthology.misc.health.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:29:52.428 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T11:29:52.437051+0000 2024-03-29T11:29:52.428 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T11:29:52.437051+0000 2024-03-29T11:29:52.428 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-29T11:29:52.428 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:29:52.429 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-29T11:29:52.429 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.noname-c 2024-03-29T11:29:52.429 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-b 2024-03-29T11:29:52.429 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:29:52.429 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:52.437+0000 7f57225e8640 1 Processor -- start 2024-03-29T11:29:52.429 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:52.437+0000 7f57225e8640 1 -- start start 2024-03-29T11:29:52.429 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:52.437+0000 7f57225e8640 10 monclient: init 2024-03-29T11:29:52.431 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:52.438+0000 7f57225e8640 10 monclient: _reopen_session rank -1 2024-03-29T11:29:52.431 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:52.438+0000 7f57225e8640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:29:52.431 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:52.438+0000 7f57225e8640 10 monclient(hunting): picked mon.noname-b con 0x7f57140389e0 addr v1:172.21.15.84:6790/0 2024-03-29T11:29:52.431 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:52.438+0000 7f57225e8640 10 monclient(hunting): picked mon.noname-c con 0x7f5714038dc0 addr v1:172.21.15.179:6789/0 2024-03-29T11:29:52.431 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:52.438+0000 7f57225e8640 10 monclient(hunting): picked mon.noname-a con 0x7f5714030660 addr v1:172.21.15.84:6789/0 2024-03-29T11:29:52.431 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:52.438+0000 7f57225e8640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f57140c54c0 con 0x7f5714030660 2024-03-29T11:29:52.431 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:52.438+0000 7f57225e8640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5714036430 con 0x7f5714038dc0 2024-03-29T11:29:52.431 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:52.439+0000 7f57225e8640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5714039840 con 0x7f57140389e0 2024-03-29T11:29:52.432 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:52.439+0000 7f57225e8640 10 monclient(hunting): _renew_subs 2024-03-29T11:29:52.432 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:52.439+0000 7f57225e8640 10 monclient(hunting): authenticate will time out at 1453.733254s 2024-03-29T11:29:52.432 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:52.438+0000 7f57215e6640 1 --1- >> v1:172.21.15.84:6790/0 conn(0x7f57140389e0 0x7f5714033520 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.84:6790/0 says I am v1:172.21.15.84:34214/0 (socket says 172.21.15.84:34214) 2024-03-29T11:29:52.432 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:52.439+0000 7f57215e6640 1 -- 172.21.15.84:0/1160250093 learned_addr learned my addr 172.21.15.84:0/1160250093 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-29T11:29:52.433 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:52.441+0000 7f5713fff640 1 -- 172.21.15.84:0/1160250093 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 88336571 0 0) 0x7f5714039840 con 0x7f57140389e0 2024-03-29T11:29:52.433 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:52.442+0000 7f5713fff640 10 monclient(hunting): _negotiate 2024-03-29T11:29:52.433 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:52.442+0000 7f5713fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:29:52.434 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:52.442+0000 7f5713fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:29:52.434 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:52.442+0000 7f5713fff640 10 monclient(hunting): my global_id is 4661 2024-03-29T11:29:52.434 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:52.442+0000 7f5713fff640 1 -- 172.21.15.84:0/1160250093 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5708003520 con 0x7f57140389e0 2024-03-29T11:29:52.434 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:52.442+0000 7f5713fff640 1 -- 172.21.15.84:0/1160250093 <== mon.2 v1:172.21.15.84:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3917841662 0 0) 0x7f5708003520 con 0x7f57140389e0 2024-03-29T11:29:52.434 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:52.442+0000 7f5713fff640 1 -- 172.21.15.84:0/1160250093 >> v1:172.21.15.179:6789/0 conn(0x7f5714038dc0 legacy=0x7f57140cadd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:29:52.435 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:52.443+0000 7f5713fff640 1 -- 172.21.15.84:0/1160250093 >> v1:172.21.15.84:6789/0 conn(0x7f5714030660 legacy=0x7f57140d3170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:52.435 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:52.443+0000 7f5713fff640 10 monclient: _finish_hunting 0 2024-03-29T11:29:52.435 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:52.443+0000 7f5713fff640 1 monclient: found mon.noname-b 2024-03-29T11:29:52.435 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:52.443+0000 7f5713fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:29:52.435 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:52.443+0000 7f5713fff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.84:6790/0 2024-03-29T11:29:52.435 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:52.443+0000 7f5713fff640 1 -- 172.21.15.84:0/1160250093 --> v1:172.21.15.84:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5714035690 con 0x7f57140389e0 2024-03-29T11:29:52.435 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:52.443+0000 7f5713fff640 10 monclient: _finish_auth 0 2024-03-29T11:29:52.435 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:52.443+0000 7f5713fff640 10 monclient: _check_auth_tickets 2024-03-29T11:29:52.435 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:52.443+0000 7f5713fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:29:52.435 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:52.443+0000 7f5713fff640 1 -- 172.21.15.84:0/1160250093 <== mon.2 v1:172.21.15.84:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4236868694 0 0) 0x7f5718001690 con 0x7f57140389e0 2024-03-29T11:29:52.436 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:52.443+0000 7f5713fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:29:52.436 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:52.443+0000 7f5713fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:29:52.436 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:52.443+0000 7f5713fff640 10 monclient: dump: 2024-03-29T11:29:52.436 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T11:29:52.436 INFO:teuthology.misc.health.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:29:52.436 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T11:18:53.864025+0000 2024-03-29T11:29:52.436 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T11:18:53.864025+0000 2024-03-29T11:29:52.436 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T11:29:52.436 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:29:52.436 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T11:29:52.436 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T11:29:52.436 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T11:29:52.436 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:29:52.436 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:52.443+0000 7f57225e8640 5 monclient: authenticate success, global_id 4661 2024-03-29T11:29:52.437 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:52.443+0000 7f57225e8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:29:52.437 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:52.443+0000 7f5713fff640 1 -- 172.21.15.84:0/1160250093 <== mon.2 v1:172.21.15.84:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5718001920 con 0x7f57140389e0 2024-03-29T11:29:52.437 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:52.443+0000 7f5713fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:29:52.437 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:52.443+0000 7f5713fff640 1 -- 172.21.15.84:0/1160250093 <== mon.2 v1:172.21.15.84:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4236868694 0 0) 0x7f5718001cb0 con 0x7f57140389e0 2024-03-29T11:29:52.438 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:52.443+0000 7f5713fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:29:52.438 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:52.443+0000 7f5713fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:29:52.438 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:52.443+0000 7f5713fff640 10 monclient: dump: 2024-03-29T11:29:52.438 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T11:29:52.438 INFO:teuthology.misc.health.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:29:52.438 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T11:18:53.864025+0000 2024-03-29T11:29:52.438 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T11:18:53.864025+0000 2024-03-29T11:29:52.438 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T11:29:52.438 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:29:52.438 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T11:29:52.438 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T11:29:52.439 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T11:29:52.439 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:29:52.439 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:52.443+0000 7f57225e8640 10 monclient: get_monmap_and_config success 2024-03-29T11:29:52.439 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:52.443+0000 7f57225e8640 4 set_mon_vals no callback set 2024-03-29T11:29:52.439 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:52.444+0000 7f57225e8640 10 monclient: shutdown 2024-03-29T11:29:52.439 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:52.444+0000 7f57225e8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:29:52.439 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:52.444+0000 7f57225e8640 1 -- 172.21.15.84:0/1160250093 >> v1:172.21.15.84:6790/0 conn(0x7f57140389e0 legacy=0x7f5714033520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:52.439 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:52.444+0000 7f57225e8640 1 -- 172.21.15.84:0/1160250093 shutdown_connections 2024-03-29T11:29:52.439 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:52.444+0000 7f57225e8640 1 -- 172.21.15.84:0/1160250093 >> 172.21.15.84:0/1160250093 conn(0x7f5714005c00 msgr2=0x7f5714006010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:29:52.439 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:52.446+0000 7f57225e8640 1 -- 172.21.15.84:0/1160250093 shutdown_connections 2024-03-29T11:29:52.441 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:52.446+0000 7f57225e8640 1 -- 172.21.15.84:0/1160250093 wait complete. 2024-03-29T11:29:52.441 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:52.447+0000 7f57225e8640 10 monclient: build_initial_monmap 2024-03-29T11:29:52.441 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:52.447+0000 7f57225e8640 10 monclient: monmap: 2024-03-29T11:29:52.441 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-29T11:29:52.441 INFO:teuthology.misc.health.smithi084.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:29:52.441 INFO:teuthology.misc.health.smithi084.stderr:last_changed 0.000000 2024-03-29T11:29:52.441 INFO:teuthology.misc.health.smithi084.stderr:created 0.000000 2024-03-29T11:29:52.441 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-29T11:29:52.441 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:29:52.441 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-29T11:29:52.441 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.noname-b 2024-03-29T11:29:52.441 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-c 2024-03-29T11:29:52.441 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:29:52.441 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:52.447+0000 7f57225e8640 1 librados: starting msgr at 2024-03-29T11:29:52.442 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:52.447+0000 7f57225e8640 1 librados: starting objecter 2024-03-29T11:29:52.442 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:52.447+0000 7f57225e8640 1 Processor -- start 2024-03-29T11:29:52.442 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:52.447+0000 7f57225e8640 1 -- start start 2024-03-29T11:29:52.442 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:52.447+0000 7f57225e8640 1 librados: setting wanted keys 2024-03-29T11:29:52.442 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:52.447+0000 7f57225e8640 1 librados: calling monclient init 2024-03-29T11:29:52.442 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:52.447+0000 7f57225e8640 10 monclient: init 2024-03-29T11:29:52.442 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:52.448+0000 7f57225e8640 10 monclient: _reopen_session rank -1 2024-03-29T11:29:52.442 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:52.448+0000 7f57225e8640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:29:52.443 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:52.451+0000 7f57225e8640 10 monclient(hunting): picked mon.noname-b con 0x7f5714030660 addr v1:172.21.15.179:6789/0 2024-03-29T11:29:52.443 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:52.451+0000 7f57215e6640 1 --1- >> v1:172.21.15.179:6789/0 conn(0x7f5714030660 0x7f5714169970 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.179:6789/0 says I am v1:172.21.15.84:35683/0 (socket says 172.21.15.84:35683) 2024-03-29T11:29:52.443 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:52.451+0000 7f57215e6640 1 -- 172.21.15.84:0/2273852347 learned_addr learned my addr 172.21.15.84:0/2273852347 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-29T11:29:52.443 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:52.451+0000 7f57225e8640 10 monclient(hunting): picked mon.noname-c con 0x7f57140389e0 addr v1:172.21.15.84:6790/0 2024-03-29T11:29:52.443 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:52.451+0000 7f5711ffb640 10 client.?.objecter ms_handle_connect 0x7f5714030660 2024-03-29T11:29:52.443 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:52.451+0000 7f5711ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:29:52.450 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:52.451+0000 7f57225e8640 10 monclient(hunting): picked mon.noname-a con 0x7f571402e750 addr v1:172.21.15.84:6789/0 2024-03-29T11:29:52.450 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:52.451+0000 7f57225e8640 1 -- 172.21.15.84:0/2273852347 --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f571402f7f0 con 0x7f571402e750 2024-03-29T11:29:52.450 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:52.451+0000 7f57225e8640 1 -- 172.21.15.84:0/2273852347 --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5714166f70 con 0x7f5714030660 2024-03-29T11:29:52.450 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:52.451+0000 7f57225e8640 1 -- 172.21.15.84:0/2273852347 --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f571402fa40 con 0x7f57140389e0 2024-03-29T11:29:52.450 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:52.451+0000 7f57225e8640 10 monclient(hunting): _renew_subs 2024-03-29T11:29:52.450 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:52.451+0000 7f57225e8640 10 monclient(hunting): authenticate will time out at 1453.745930s 2024-03-29T11:29:52.450 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:52.451+0000 7f5711ffb640 10 client.?.objecter ms_handle_connect 0x7f57140389e0 2024-03-29T11:29:52.450 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:52.451+0000 7f5711ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:29:52.451 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:52.452+0000 7f5711ffb640 1 -- 172.21.15.84:0/2273852347 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1133513465 0 0) 0x7f571402fa40 con 0x7f57140389e0 2024-03-29T11:29:52.451 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:52.452+0000 7f5711ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:29:52.451 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:52.452+0000 7f5711ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:29:52.451 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:52.452+0000 7f5711ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:29:52.451 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:52.452+0000 7f5711ffb640 10 monclient(hunting): my global_id is 4664 2024-03-29T11:29:52.451 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:52.452+0000 7f5711ffb640 1 -- 172.21.15.84:0/2273852347 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5700004eb0 con 0x7f57140389e0 2024-03-29T11:29:52.451 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:52.452+0000 7f5711ffb640 1 -- 172.21.15.84:0/2273852347 <== mon.2 v1:172.21.15.84:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3688249474 0 0) 0x7f5700004eb0 con 0x7f57140389e0 2024-03-29T11:29:52.451 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:52.452+0000 7f5711ffb640 1 -- 172.21.15.84:0/2273852347 --> v1:172.21.15.84:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f571402fa40 con 0x7f57140389e0 2024-03-29T11:29:52.451 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:52.452+0000 7f5711ffb640 1 -- 172.21.15.84:0/2273852347 <== mon.2 v1:172.21.15.84:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4236868694 0 0) 0x7f570c0017d0 con 0x7f57140389e0 2024-03-29T11:29:52.451 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:52.452+0000 7f5711ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:29:52.451 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:52.452+0000 7f5711ffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:29:52.451 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:52.452+0000 7f5711ffb640 10 monclient(hunting): dump: 2024-03-29T11:29:52.451 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T11:29:52.452 INFO:teuthology.misc.health.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:29:52.452 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T11:18:53.864025+0000 2024-03-29T11:29:52.452 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T11:18:53.864025+0000 2024-03-29T11:29:52.452 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T11:29:52.452 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:29:52.452 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T11:29:52.452 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T11:29:52.452 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T11:29:52.452 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:29:52.452 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:52.452+0000 7f5711ffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:29:52.453 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:52.453+0000 7f5711ffb640 1 -- 172.21.15.84:0/2273852347 <== mon.2 v1:172.21.15.84:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 602425271 0 0) 0x7f571402fa40 con 0x7f57140389e0 2024-03-29T11:29:52.453 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:52.453+0000 7f5711ffb640 1 -- 172.21.15.84:0/2273852347 >> v1:172.21.15.179:6789/0 conn(0x7f5714030660 legacy=0x7f5714169970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:52.453 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:52.453+0000 7f5711ffb640 1 -- 172.21.15.84:0/2273852347 >> v1:172.21.15.84:6789/0 conn(0x7f571402e750 legacy=0x7f5714166860 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:29:52.453 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:52.454+0000 7f5711ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:29:52.453 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:52.454+0000 7f5711ffb640 1 monclient: found mon.c 2024-03-29T11:29:52.453 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:52.454+0000 7f5711ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:29:52.453 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:52.454+0000 7f5711ffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-29T11:29:52.453 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:52.454+0000 7f5711ffb640 1 -- 172.21.15.84:0/2273852347 --> v1:172.21.15.84:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f571402fd20 con 0x7f57140389e0 2024-03-29T11:29:52.453 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:52.455+0000 7f5711ffb640 10 monclient: _finish_auth 0 2024-03-29T11:29:52.453 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:52.455+0000 7f5711ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:29:52.453 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:52.455+0000 7f5711ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:29:52.453 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:52.455+0000 7f57225e8640 5 monclient: authenticate success, global_id 4664 2024-03-29T11:29:52.453 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:52.455+0000 7f57225e8640 10 monclient: _renew_subs 2024-03-29T11:29:52.453 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:52.455+0000 7f57225e8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-29T11:29:52.453 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:52.455+0000 7f57225e8640 1 -- 172.21.15.84:0/2273852347 --> v1:172.21.15.84:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f571402eea0 con 0x7f57140389e0 2024-03-29T11:29:52.454 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:52.455+0000 7f57225e8640 10 client.4664.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:29:52.454 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:52.455+0000 7f57225e8640 10 monclient: _renew_subs 2024-03-29T11:29:52.454 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:52.455+0000 7f57225e8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-29T11:29:52.454 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:52.455+0000 7f57225e8640 1 -- 172.21.15.84:0/2273852347 --> v1:172.21.15.84:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f57140c67c0 con 0x7f57140389e0 2024-03-29T11:29:52.454 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:52.455+0000 7f57225e8640 1 librados: init done 2024-03-29T11:29:52.454 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:52.456+0000 7f5711ffb640 1 -- 172.21.15.84:0/2273852347 <== mon.2 v1:172.21.15.84:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f570c001d60 con 0x7f57140389e0 2024-03-29T11:29:52.454 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:52.456+0000 7f5711ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:29:52.454 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:52.456+0000 7f5711ffb640 1 -- 172.21.15.84:0/2273852347 <== mon.2 v1:172.21.15.84:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4236868694 0 0) 0x7f570c003790 con 0x7f57140389e0 2024-03-29T11:29:52.454 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:52.456+0000 7f5711ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:29:52.454 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:52.456+0000 7f5711ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:29:52.455 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:52.456+0000 7f5711ffb640 10 monclient: dump: 2024-03-29T11:29:52.455 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T11:29:52.455 INFO:teuthology.misc.health.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:29:52.455 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T11:18:53.864025+0000 2024-03-29T11:29:52.456 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T11:18:53.864025+0000 2024-03-29T11:29:52.456 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T11:29:52.456 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:29:52.456 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T11:29:52.456 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T11:29:52.456 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T11:29:52.456 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:29:52.456 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:52.456+0000 7f57137fe640 4 set_mon_vals no callback set 2024-03-29T11:29:52.456 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:52.457+0000 7f57077fe640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:29:52.456 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:52.457+0000 7f57077fe640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:29:52.456 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:52.457+0000 7f57077fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-29T11:29:52.456 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:52.457+0000 7f57077fe640 1 -- 172.21.15.84:0/2273852347 --> v1:172.21.15.84:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f56f4005080 con 0x7f57140389e0 2024-03-29T11:29:52.457 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:52.458+0000 7f5711ffb640 1 -- 172.21.15.84:0/2273852347 <== mon.2 v1:172.21.15.84:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4033942909 0 0) 0x7f570c0039f0 con 0x7f57140389e0 2024-03-29T11:29:52.458 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:52.459+0000 7f5711ffb640 1 -- 172.21.15.84:0/2273852347 <== mon.2 v1:172.21.15.84:6790/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 156031568 0 0) 0x7f570c0836a0 con 0x7f57140389e0 2024-03-29T11:29:52.458 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:52.459+0000 7f5711ffb640 10 client.4664.objecter ms_dispatch 0x7f571c000b90 osd_map(11..11 src has 1..11) v4 2024-03-29T11:29:52.458 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:52.459+0000 7f5711ffb640 3 client.4664.objecter handle_osd_map got epochs [11,11] > 0 2024-03-29T11:29:52.458 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:52.459+0000 7f5711ffb640 3 client.4664.objecter handle_osd_map decoding full epoch 11 2024-03-29T11:29:52.458 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:52.459+0000 7f5711ffb640 20 client.4664.objecter dump_active .. 0 homeless 2024-03-29T11:29:52.458 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:52.461+0000 7f57215e6640 10 monclient: get_auth_request con 0x7f570006dc10 auth_method 0 2024-03-29T11:29:52.458 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:52.463+0000 7f5711ffb640 10 client.4664.objecter ms_handle_connect 0x7f570006dc10 2024-03-29T11:29:52.458 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:52.466+0000 7f5711ffb640 1 -- 172.21.15.84:0/2273852347 <== mon.2 v1:172.21.15.84:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f570c04ef80 con 0x7f57140389e0 2024-03-29T11:29:52.458 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:52.466+0000 7f5711ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:29:52.459 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:52.466+0000 7f5711ffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:29:52.786 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:52.794+0000 7f57077fe640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:29:52.786 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:52.794+0000 7f57077fe640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:29:52.786 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:52.794+0000 7f57077fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-29T11:29:52.786 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:52.794+0000 7f57077fe640 1 -- 172.21.15.84:0/2273852347 --> v1:172.21.15.84:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f56f4005330 con 0x7f57140389e0 2024-03-29T11:29:52.787 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:52.795+0000 7f5711ffb640 1 -- 172.21.15.84:0/2273852347 <== mon.2 v1:172.21.15.84:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 782149409) 0x7f570c050eb0 con 0x7f57140389e0 2024-03-29T11:29:52.787 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:52.795+0000 7f5711ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:29:52.787 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:52.795+0000 7f5711ffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:29:52.788 INFO:teuthology.misc.health.smithi084.stdout: 2024-03-29T11:29:52.788 INFO:teuthology.misc.health.smithi084.stdout:{"epoch":11,"fsid":"2c3e002f-9eb0-4860-853b-4ef1ef562213","created":"2024-03-29T11:19:32.029169+0000","modified":"2024-03-29T11:29:36.955202+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:29:36.955202+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"5c39e276-2034-4716-bbc0-57a634eefa92","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"cd580e99-109f-4bf1-88c0-a5987bc8b9ed","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"11e60a5a-051d-453f-a190-66387cb35592","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"2c329fbe-bbc4-4030-bf87-d711cad04679","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f4213d4a-bd36-4433-91ae-d7e2b02481d3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"f0cf44ea-6055-48af-805b-aed858102ff9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"d895eaa0-cc0b-4ee4-9b5a-2dfcd89a60b7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"9bc5a7bc-8f52-4db9-96d6-de5e68d52eaa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:29:52.791 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:52.800+0000 7f57077fe640 10 librados: watch_flush enter 2024-03-29T11:29:52.794 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:52.801+0000 7f57077fe640 10 librados: watch_flush exit 2024-03-29T11:29:52.794 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:52.801+0000 7f57077fe640 20 client.4664.objecter shutdown clearing up homeless session... 2024-03-29T11:29:52.794 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:52.801+0000 7f57077fe640 10 client.4664.objecter successfully canceled tick 2024-03-29T11:29:52.794 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:52.802+0000 7f57077fe640 1 -- 172.21.15.84:0/2273852347 >> v1:172.21.15.179:6816/2981134980 conn(0x7f570006dc10 legacy=0x7f57000700d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:52.794 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:52.802+0000 7f57077fe640 10 monclient: shutdown 2024-03-29T11:29:52.794 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:52.802+0000 7f57077fe640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:29:52.794 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:52.802+0000 7f57077fe640 1 -- 172.21.15.84:0/2273852347 >> v1:172.21.15.84:6790/0 conn(0x7f57140389e0 legacy=0x7f5714166150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:52.794 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:52.803+0000 7f57077fe640 1 -- 172.21.15.84:0/2273852347 shutdown_connections 2024-03-29T11:29:52.794 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:52.803+0000 7f57077fe640 1 -- 172.21.15.84:0/2273852347 >> 172.21.15.84:0/2273852347 conn(0x7f5714005c00 msgr2=0x7f5714034750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:29:52.795 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:52.803+0000 7f57077fe640 1 -- 172.21.15.84:0/2273852347 shutdown_connections 2024-03-29T11:29:52.795 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:52.804+0000 7f57077fe640 1 -- 172.21.15.84:0/2273852347 wait complete. 2024-03-29T11:29:52.796 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:52.804+0000 7f57077fe640 1 librados: shutdown 2024-03-29T11:29:52.813 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:29:59.620 DEBUG:teuthology.orchestra.run.smithi084:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:29:59.761 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:59.769+0000 7fe232a7b640 10 monclient: get_monmap_and_config 2024-03-29T11:29:59.761 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:59.769+0000 7fe232a7b640 10 monclient: build_initial_monmap 2024-03-29T11:29:59.761 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:59.769+0000 7fe232a7b640 10 monclient: monmap: 2024-03-29T11:29:59.761 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-29T11:29:59.761 INFO:teuthology.misc.health.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:29:59.761 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T11:29:59.770290+0000 2024-03-29T11:29:59.761 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T11:29:59.770290+0000 2024-03-29T11:29:59.762 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-29T11:29:59.762 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:29:59.762 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-29T11:29:59.762 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.noname-c 2024-03-29T11:29:59.762 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-b 2024-03-29T11:29:59.762 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:29:59.762 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:59.770+0000 7fe232a7b640 1 Processor -- start 2024-03-29T11:29:59.763 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:59.770+0000 7fe232a7b640 1 -- start start 2024-03-29T11:29:59.763 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:59.770+0000 7fe232a7b640 10 monclient: init 2024-03-29T11:29:59.766 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:59.771+0000 7fe232a7b640 10 monclient: _reopen_session rank -1 2024-03-29T11:29:59.766 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:59.771+0000 7fe232a7b640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:29:59.766 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:59.771+0000 7fe232a7b640 10 monclient(hunting): picked mon.noname-a con 0x7fe22c1457f0 addr v1:172.21.15.84:6789/0 2024-03-29T11:29:59.766 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:59.771+0000 7fe232a7b640 10 monclient(hunting): picked mon.noname-b con 0x7fe22c13f650 addr v1:172.21.15.84:6790/0 2024-03-29T11:29:59.766 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:59.771+0000 7fe232a7b640 10 monclient(hunting): picked mon.noname-c con 0x7fe22c0a9d70 addr v1:172.21.15.179:6789/0 2024-03-29T11:29:59.766 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:59.771+0000 7fe232a7b640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe22c0aaef0 con 0x7fe22c1457f0 2024-03-29T11:29:59.767 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:59.771+0000 7fe232a7b640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe22c145bd0 con 0x7fe22c0a9d70 2024-03-29T11:29:59.767 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:59.771+0000 7fe232a7b640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe22c145eb0 con 0x7fe22c13f650 2024-03-29T11:29:59.767 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:59.771+0000 7fe232a7b640 10 monclient(hunting): _renew_subs 2024-03-29T11:29:59.767 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:59.771+0000 7fe232a7b640 10 monclient(hunting): authenticate will time out at 1461.065599s 2024-03-29T11:29:59.767 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:59.771+0000 7fe230817640 1 --1- >> v1:172.21.15.84:6789/0 conn(0x7fe22c1457f0 0x7fe22c13ef40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.84:6789/0 says I am v1:172.21.15.84:47810/0 (socket says 172.21.15.84:47810) 2024-03-29T11:29:59.767 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:59.771+0000 7fe230817640 1 -- 172.21.15.84:0/824796402 learned_addr learned my addr 172.21.15.84:0/824796402 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-29T11:29:59.767 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:59.772+0000 7fe22b7fe640 1 -- 172.21.15.84:0/824796402 <== mon.0 v1:172.21.15.84:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2858069799 0 0) 0x7fe22c0aaef0 con 0x7fe22c1457f0 2024-03-29T11:29:59.767 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:59.772+0000 7fe22b7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:29:59.767 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:59.772+0000 7fe22b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:29:59.767 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:59.772+0000 7fe22b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:29:59.767 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:59.772+0000 7fe22b7fe640 10 monclient(hunting): my global_id is 4590 2024-03-29T11:29:59.767 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:59.772+0000 7fe22b7fe640 1 -- 172.21.15.84:0/824796402 --> v1:172.21.15.84:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe218003520 con 0x7fe22c1457f0 2024-03-29T11:29:59.767 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:59.772+0000 7fe22b7fe640 1 -- 172.21.15.84:0/824796402 <== mon.0 v1:172.21.15.84:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 282864914 0 0) 0x7fe218003520 con 0x7fe22c1457f0 2024-03-29T11:29:59.768 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:59.772+0000 7fe22b7fe640 1 -- 172.21.15.84:0/824796402 >> v1:172.21.15.84:6790/0 conn(0x7fe22c13f650 legacy=0x7fe22c0b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:59.768 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:59.772+0000 7fe22b7fe640 1 -- 172.21.15.84:0/824796402 >> v1:172.21.15.179:6789/0 conn(0x7fe22c0a9d70 legacy=0x7fe22c14d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:59.768 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:59.772+0000 7fe22b7fe640 10 monclient: _finish_hunting 0 2024-03-29T11:29:59.768 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:59.772+0000 7fe22b7fe640 1 monclient: found mon.noname-a 2024-03-29T11:29:59.768 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:59.772+0000 7fe22b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:29:59.768 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:59.772+0000 7fe22b7fe640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.84:6789/0 2024-03-29T11:29:59.768 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:59.772+0000 7fe22b7fe640 1 -- 172.21.15.84:0/824796402 --> v1:172.21.15.84:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe22c146340 con 0x7fe22c1457f0 2024-03-29T11:29:59.768 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:59.772+0000 7fe22b7fe640 10 monclient: _finish_auth 0 2024-03-29T11:29:59.768 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:59.772+0000 7fe22b7fe640 10 monclient: _check_auth_tickets 2024-03-29T11:29:59.768 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:59.772+0000 7fe22b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:29:59.768 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:59.772+0000 7fe22b7fe640 1 -- 172.21.15.84:0/824796402 <== mon.0 v1:172.21.15.84:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4236868694 0 0) 0x7fe224001a10 con 0x7fe22c1457f0 2024-03-29T11:29:59.768 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:59.772+0000 7fe22b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:29:59.768 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:59.772+0000 7fe22b7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:29:59.768 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:59.772+0000 7fe22b7fe640 10 monclient: dump: 2024-03-29T11:29:59.768 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T11:29:59.769 INFO:teuthology.misc.health.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:29:59.769 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T11:18:53.864025+0000 2024-03-29T11:29:59.769 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T11:18:53.864025+0000 2024-03-29T11:29:59.769 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T11:29:59.769 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:29:59.769 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T11:29:59.769 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T11:29:59.769 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T11:29:59.769 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:29:59.769 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:59.772+0000 7fe232a7b640 5 monclient: authenticate success, global_id 4590 2024-03-29T11:29:59.769 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:59.772+0000 7fe232a7b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:29:59.769 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:59.773+0000 7fe22b7fe640 1 -- 172.21.15.84:0/824796402 <== mon.0 v1:172.21.15.84:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe224001b70 con 0x7fe22c1457f0 2024-03-29T11:29:59.769 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:59.773+0000 7fe22b7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:29:59.769 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:59.773+0000 7fe22b7fe640 1 -- 172.21.15.84:0/824796402 <== mon.0 v1:172.21.15.84:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4236868694 0 0) 0x7fe224001fc0 con 0x7fe22c1457f0 2024-03-29T11:29:59.769 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:59.773+0000 7fe22b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:29:59.770 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:59.773+0000 7fe22b7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:29:59.770 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:59.773+0000 7fe22b7fe640 10 monclient: dump: 2024-03-29T11:29:59.770 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T11:29:59.770 INFO:teuthology.misc.health.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:29:59.770 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T11:18:53.864025+0000 2024-03-29T11:29:59.770 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T11:18:53.864025+0000 2024-03-29T11:29:59.770 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T11:29:59.770 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:29:59.770 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T11:29:59.770 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T11:29:59.770 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T11:29:59.770 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:29:59.770 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:59.773+0000 7fe232a7b640 10 monclient: get_monmap_and_config success 2024-03-29T11:29:59.770 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:59.773+0000 7fe232a7b640 4 set_mon_vals no callback set 2024-03-29T11:29:59.771 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:59.774+0000 7fe232a7b640 10 monclient: shutdown 2024-03-29T11:29:59.771 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:59.774+0000 7fe232a7b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:29:59.771 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:59.774+0000 7fe232a7b640 1 -- 172.21.15.84:0/824796402 >> v1:172.21.15.84:6789/0 conn(0x7fe22c1457f0 legacy=0x7fe22c13ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:59.771 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:59.774+0000 7fe232a7b640 1 -- 172.21.15.84:0/824796402 shutdown_connections 2024-03-29T11:29:59.771 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:59.774+0000 7fe232a7b640 1 -- 172.21.15.84:0/824796402 >> 172.21.15.84:0/824796402 conn(0x7fe22c066a00 msgr2=0x7fe22c066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:29:59.771 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:59.774+0000 7fe232a7b640 1 -- 172.21.15.84:0/824796402 shutdown_connections 2024-03-29T11:29:59.771 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:59.774+0000 7fe232a7b640 1 -- 172.21.15.84:0/824796402 wait complete. 2024-03-29T11:29:59.777 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:59.775+0000 7fe232a7b640 10 monclient: build_initial_monmap 2024-03-29T11:29:59.777 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:59.775+0000 7fe232a7b640 10 monclient: monmap: 2024-03-29T11:29:59.777 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-29T11:29:59.778 INFO:teuthology.misc.health.smithi084.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:29:59.778 INFO:teuthology.misc.health.smithi084.stderr:last_changed 0.000000 2024-03-29T11:29:59.778 INFO:teuthology.misc.health.smithi084.stderr:created 0.000000 2024-03-29T11:29:59.778 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-29T11:29:59.778 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:29:59.778 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-29T11:29:59.778 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.noname-b 2024-03-29T11:29:59.778 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-c 2024-03-29T11:29:59.778 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:29:59.778 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:59.776+0000 7fe232a7b640 1 librados: starting msgr at 2024-03-29T11:29:59.778 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:59.776+0000 7fe232a7b640 1 librados: starting objecter 2024-03-29T11:29:59.778 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:59.776+0000 7fe232a7b640 1 Processor -- start 2024-03-29T11:29:59.778 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:59.776+0000 7fe232a7b640 1 -- start start 2024-03-29T11:29:59.778 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:59.776+0000 7fe232a7b640 1 librados: setting wanted keys 2024-03-29T11:29:59.779 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:59.776+0000 7fe232a7b640 1 librados: calling monclient init 2024-03-29T11:29:59.779 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:59.776+0000 7fe232a7b640 10 monclient: init 2024-03-29T11:29:59.779 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:59.777+0000 7fe232a7b640 10 monclient: _reopen_session rank -1 2024-03-29T11:29:59.779 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:59.777+0000 7fe232a7b640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:29:59.779 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:59.777+0000 7fe232a7b640 10 monclient(hunting): picked mon.noname-a con 0x7fe22c0a9d70 addr v1:172.21.15.84:6789/0 2024-03-29T11:29:59.779 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:59.777+0000 7fe232a7b640 10 monclient(hunting): picked mon.noname-b con 0x7fe22c13f650 addr v1:172.21.15.179:6789/0 2024-03-29T11:29:59.779 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:59.777+0000 7fe232a7b640 10 monclient(hunting): picked mon.noname-c con 0x7fe22c1457f0 addr v1:172.21.15.84:6790/0 2024-03-29T11:29:59.779 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:59.777+0000 7fe232a7b640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe22c144f70 con 0x7fe22c0a9d70 2024-03-29T11:29:59.779 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:59.777+0000 7fe232a7b640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe22c144c00 con 0x7fe22c13f650 2024-03-29T11:29:59.779 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:59.777+0000 7fe232a7b640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe22c137800 con 0x7fe22c1457f0 2024-03-29T11:29:59.779 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:59.777+0000 7fe232a7b640 10 monclient(hunting): _renew_subs 2024-03-29T11:29:59.779 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:59.777+0000 7fe232a7b640 10 monclient(hunting): authenticate will time out at 1461.071906s 2024-03-29T11:29:59.779 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:59.777+0000 7fe230817640 1 --1- >> v1:172.21.15.84:6789/0 conn(0x7fe22c0a9d70 0x7fe22c142590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.84:6789/0 says I am v1:172.21.15.84:49000/0 (socket says 172.21.15.84:49000) 2024-03-29T11:29:59.779 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:59.777+0000 7fe230817640 1 -- 172.21.15.84:0/978184157 learned_addr learned my addr 172.21.15.84:0/978184157 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-29T11:29:59.780 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:59.778+0000 7fe2297fa640 10 client.?.objecter ms_handle_connect 0x7fe22c0a9d70 2024-03-29T11:29:59.780 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:59.778+0000 7fe2297fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:29:59.780 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:59.778+0000 7fe2297fa640 10 client.?.objecter ms_handle_connect 0x7fe22c13f650 2024-03-29T11:29:59.780 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:59.778+0000 7fe2297fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:29:59.780 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:59.778+0000 7fe2297fa640 10 client.?.objecter ms_handle_connect 0x7fe22c1457f0 2024-03-29T11:29:59.780 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:59.778+0000 7fe2297fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:29:59.780 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:59.778+0000 7fe2297fa640 1 -- 172.21.15.84:0/978184157 <== mon.0 v1:172.21.15.84:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2400836905 0 0) 0x7fe22c144f70 con 0x7fe22c0a9d70 2024-03-29T11:29:59.780 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:59.778+0000 7fe2297fa640 10 monclient(hunting): _negotiate 2024-03-29T11:29:59.780 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:59.778+0000 7fe2297fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:29:59.780 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:59.778+0000 7fe2297fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:29:59.780 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:59.778+0000 7fe2297fa640 10 monclient(hunting): my global_id is 4593 2024-03-29T11:29:59.780 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:59.778+0000 7fe2297fa640 1 -- 172.21.15.84:0/978184157 --> v1:172.21.15.84:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe20c004eb0 con 0x7fe22c0a9d70 2024-03-29T11:29:59.780 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:59.778+0000 7fe2297fa640 1 -- 172.21.15.84:0/978184157 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1470523848 0 0) 0x7fe22c137800 con 0x7fe22c1457f0 2024-03-29T11:29:59.780 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:59.778+0000 7fe2297fa640 10 monclient(hunting): _negotiate 2024-03-29T11:29:59.780 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:59.778+0000 7fe2297fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:29:59.781 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:59.778+0000 7fe2297fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:29:59.781 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:59.778+0000 7fe2297fa640 10 monclient(hunting): my global_id is 4670 2024-03-29T11:29:59.781 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:59.778+0000 7fe2297fa640 1 -- 172.21.15.84:0/978184157 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe22c144f70 con 0x7fe22c1457f0 2024-03-29T11:29:59.781 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:59.779+0000 7fe2297fa640 1 -- 172.21.15.84:0/978184157 <== mon.2 v1:172.21.15.84:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1353685720 0 0) 0x7fe22c144f70 con 0x7fe22c1457f0 2024-03-29T11:29:59.781 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:59.779+0000 7fe2297fa640 1 -- 172.21.15.84:0/978184157 --> v1:172.21.15.84:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe22c137800 con 0x7fe22c1457f0 2024-03-29T11:29:59.781 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:59.779+0000 7fe2297fa640 1 -- 172.21.15.84:0/978184157 <== mon.2 v1:172.21.15.84:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4236868694 0 0) 0x7fe214001830 con 0x7fe22c1457f0 2024-03-29T11:29:59.781 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:59.779+0000 7fe2297fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:29:59.781 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:59.779+0000 7fe2297fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:29:59.781 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:59.779+0000 7fe2297fa640 10 monclient(hunting): dump: 2024-03-29T11:29:59.781 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T11:29:59.781 INFO:teuthology.misc.health.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:29:59.781 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T11:18:53.864025+0000 2024-03-29T11:29:59.781 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T11:18:53.864025+0000 2024-03-29T11:29:59.781 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T11:29:59.781 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:29:59.782 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T11:29:59.782 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T11:29:59.782 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T11:29:59.782 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:29:59.782 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:59.779+0000 7fe2297fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:29:59.782 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:59.779+0000 7fe2297fa640 1 -- 172.21.15.84:0/978184157 <== mon.0 v1:172.21.15.84:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1930241921 0 0) 0x7fe20c004eb0 con 0x7fe22c0a9d70 2024-03-29T11:29:59.782 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:59.779+0000 7fe2297fa640 1 -- 172.21.15.84:0/978184157 --> v1:172.21.15.84:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe22c144f70 con 0x7fe22c0a9d70 2024-03-29T11:29:59.782 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:59.779+0000 7fe2297fa640 1 -- 172.21.15.84:0/978184157 <== mon.0 v1:172.21.15.84:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4236868694 0 0) 0x7fe224001f80 con 0x7fe22c0a9d70 2024-03-29T11:29:59.782 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:59.779+0000 7fe2297fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:29:59.782 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:59.779+0000 7fe2297fa640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:29:59.782 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:59.779+0000 7fe2297fa640 10 monclient(hunting): dump: 2024-03-29T11:29:59.782 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T11:29:59.782 INFO:teuthology.misc.health.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:29:59.782 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T11:18:53.864025+0000 2024-03-29T11:29:59.783 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T11:18:53.864025+0000 2024-03-29T11:29:59.783 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T11:29:59.783 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:29:59.783 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T11:29:59.783 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T11:29:59.783 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T11:29:59.783 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:29:59.783 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:59.779+0000 7fe2297fa640 1 -- 172.21.15.84:0/978184157 <== mon.2 v1:172.21.15.84:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 954962164 0 0) 0x7fe22c137800 con 0x7fe22c1457f0 2024-03-29T11:29:59.783 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:59.779+0000 7fe2297fa640 1 -- 172.21.15.84:0/978184157 >> v1:172.21.15.179:6789/0 conn(0x7fe22c13f650 legacy=0x7fe22c1370f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:59.783 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:59.779+0000 7fe2297fa640 1 -- 172.21.15.84:0/978184157 >> v1:172.21.15.84:6789/0 conn(0x7fe22c0a9d70 legacy=0x7fe22c142590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:59.783 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:59.779+0000 7fe2297fa640 10 monclient: _finish_hunting 0 2024-03-29T11:29:59.783 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:59.779+0000 7fe2297fa640 1 monclient: found mon.c 2024-03-29T11:29:59.783 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:59.779+0000 7fe2297fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:29:59.783 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:59.779+0000 7fe2297fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-29T11:29:59.784 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:59.779+0000 7fe2297fa640 1 -- 172.21.15.84:0/978184157 --> v1:172.21.15.84:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe22c137990 con 0x7fe22c1457f0 2024-03-29T11:29:59.784 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:59.779+0000 7fe2297fa640 10 monclient: _finish_auth 0 2024-03-29T11:29:59.784 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:59.779+0000 7fe2297fa640 10 monclient: _check_auth_tickets 2024-03-29T11:29:59.784 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:59.779+0000 7fe2297fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:29:59.784 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:59.779+0000 7fe232a7b640 5 monclient: authenticate success, global_id 4670 2024-03-29T11:29:59.784 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:59.779+0000 7fe232a7b640 10 monclient: _renew_subs 2024-03-29T11:29:59.784 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:59.779+0000 7fe232a7b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-29T11:29:59.784 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:59.779+0000 7fe232a7b640 1 -- 172.21.15.84:0/978184157 --> v1:172.21.15.84:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe22c137be0 con 0x7fe22c1457f0 2024-03-29T11:29:59.784 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:59.779+0000 7fe232a7b640 10 client.4670.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:29:59.784 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:59.779+0000 7fe232a7b640 10 monclient: _renew_subs 2024-03-29T11:29:59.784 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:59.779+0000 7fe232a7b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-29T11:29:59.784 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:59.779+0000 7fe232a7b640 1 -- 172.21.15.84:0/978184157 --> v1:172.21.15.84:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe22c0aba50 con 0x7fe22c1457f0 2024-03-29T11:29:59.784 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:59.779+0000 7fe232a7b640 1 librados: init done 2024-03-29T11:29:59.784 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:59.780+0000 7fe2297fa640 1 -- 172.21.15.84:0/978184157 <== mon.2 v1:172.21.15.84:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe214001a10 con 0x7fe22c1457f0 2024-03-29T11:29:59.785 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:59.780+0000 7fe2297fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:29:59.785 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:59.780+0000 7fe2297fa640 1 -- 172.21.15.84:0/978184157 <== mon.2 v1:172.21.15.84:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4236868694 0 0) 0x7fe2140037b0 con 0x7fe22c1457f0 2024-03-29T11:29:59.785 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:59.780+0000 7fe2297fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:29:59.785 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:59.780+0000 7fe2297fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:29:59.785 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:59.780+0000 7fe2297fa640 10 monclient: dump: 2024-03-29T11:29:59.785 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T11:29:59.785 INFO:teuthology.misc.health.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:29:59.785 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T11:18:53.864025+0000 2024-03-29T11:29:59.785 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T11:18:53.864025+0000 2024-03-29T11:29:59.785 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T11:29:59.785 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:29:59.785 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T11:29:59.785 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T11:29:59.785 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T11:29:59.785 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:29:59.786 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:59.780+0000 7fe22affd640 4 set_mon_vals no callback set 2024-03-29T11:29:59.786 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:59.782+0000 7fe232a7b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:29:59.786 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:59.782+0000 7fe232a7b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:29:59.786 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:59.782+0000 7fe232a7b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-29T11:29:59.786 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:59.782+0000 7fe232a7b640 1 -- 172.21.15.84:0/978184157 --> v1:172.21.15.84:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe210005080 con 0x7fe22c1457f0 2024-03-29T11:29:59.786 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:59.782+0000 7fe2297fa640 1 -- 172.21.15.84:0/978184157 <== mon.2 v1:172.21.15.84:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4033942909 0 0) 0x7fe214003910 con 0x7fe22c1457f0 2024-03-29T11:29:59.786 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:59.783+0000 7fe230817640 10 monclient: get_auth_request con 0x7fe20c06dee0 auth_method 0 2024-03-29T11:29:59.786 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:59.783+0000 7fe2297fa640 1 -- 172.21.15.84:0/978184157 <== mon.2 v1:172.21.15.84:6790/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 156031568 0 0) 0x7fe2140837b0 con 0x7fe22c1457f0 2024-03-29T11:29:59.786 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:59.783+0000 7fe2297fa640 10 client.4670.objecter ms_dispatch 0x7fe22c000b90 osd_map(11..11 src has 1..11) v4 2024-03-29T11:29:59.786 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:59.783+0000 7fe2297fa640 3 client.4670.objecter handle_osd_map got epochs [11,11] > 0 2024-03-29T11:29:59.786 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:59.783+0000 7fe2297fa640 3 client.4670.objecter handle_osd_map decoding full epoch 11 2024-03-29T11:29:59.786 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:59.783+0000 7fe2297fa640 20 client.4670.objecter dump_active .. 0 homeless 2024-03-29T11:29:59.786 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:59.784+0000 7fe2297fa640 10 client.4670.objecter ms_handle_connect 0x7fe20c06dee0 2024-03-29T11:29:59.787 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:59.792+0000 7fe2297fa640 1 -- 172.21.15.84:0/978184157 <== mon.2 v1:172.21.15.84:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fe21404f090 con 0x7fe22c1457f0 2024-03-29T11:29:59.787 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:59.792+0000 7fe2297fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:29:59.787 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:29:59.792+0000 7fe2297fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:30:00.122 INFO:teuthology.misc.health.smithi084.stdout: 2024-03-29T11:30:00.122 INFO:teuthology.misc.health.smithi084.stdout:{"epoch":11,"fsid":"2c3e002f-9eb0-4860-853b-4ef1ef562213","created":"2024-03-29T11:19:32.029169+0000","modified":"2024-03-29T11:29:36.955202+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:29:36.955202+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"5c39e276-2034-4716-bbc0-57a634eefa92","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"cd580e99-109f-4bf1-88c0-a5987bc8b9ed","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"11e60a5a-051d-453f-a190-66387cb35592","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"2c329fbe-bbc4-4030-bf87-d711cad04679","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f4213d4a-bd36-4433-91ae-d7e2b02481d3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"f0cf44ea-6055-48af-805b-aed858102ff9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"d895eaa0-cc0b-4ee4-9b5a-2dfcd89a60b7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"9bc5a7bc-8f52-4db9-96d6-de5e68d52eaa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:30:00.123 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:00.129+0000 7fe232a7b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:30:00.123 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:00.129+0000 7fe232a7b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:30:00.123 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:00.129+0000 7fe232a7b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-29T11:30:00.123 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:00.129+0000 7fe232a7b640 1 -- 172.21.15.84:0/978184157 --> v1:172.21.15.84:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe210005330 con 0x7fe22c1457f0 2024-03-29T11:30:00.123 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:00.129+0000 7fe2297fa640 1 -- 172.21.15.84:0/978184157 <== mon.2 v1:172.21.15.84:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 782149409) 0x7fe214050fc0 con 0x7fe22c1457f0 2024-03-29T11:30:00.123 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:00.129+0000 7fe2297fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:30:00.123 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:00.129+0000 7fe2297fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:30:00.125 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:00.133+0000 7fe232a7b640 10 librados: watch_flush enter 2024-03-29T11:30:00.125 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:00.133+0000 7fe232a7b640 10 librados: watch_flush exit 2024-03-29T11:30:00.125 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:00.133+0000 7fe232a7b640 20 client.4670.objecter shutdown clearing up homeless session... 2024-03-29T11:30:00.125 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:00.133+0000 7fe232a7b640 10 client.4670.objecter successfully canceled tick 2024-03-29T11:30:00.125 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:00.133+0000 7fe232a7b640 1 -- 172.21.15.84:0/978184157 >> v1:172.21.15.179:6816/2981134980 conn(0x7fe20c06dee0 legacy=0x7fe20c0703a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:30:00.125 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:00.133+0000 7fe232a7b640 10 monclient: shutdown 2024-03-29T11:30:00.125 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:00.133+0000 7fe232a7b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:30:00.125 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:00.133+0000 7fe232a7b640 1 -- 172.21.15.84:0/978184157 >> v1:172.21.15.84:6790/0 conn(0x7fe22c1457f0 legacy=0x7fe22c1d7860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:30:00.127 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:00.135+0000 7fe232a7b640 1 -- 172.21.15.84:0/978184157 shutdown_connections 2024-03-29T11:30:00.127 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:00.135+0000 7fe232a7b640 1 -- 172.21.15.84:0/978184157 >> 172.21.15.84:0/978184157 conn(0x7fe22c066a00 msgr2=0x7fe22c13aaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:30:00.127 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:00.135+0000 7fe232a7b640 1 -- 172.21.15.84:0/978184157 shutdown_connections 2024-03-29T11:30:00.127 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:00.135+0000 7fe232a7b640 1 -- 172.21.15.84:0/978184157 wait complete. 2024-03-29T11:30:00.127 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:00.135+0000 7fe232a7b640 1 librados: shutdown 2024-03-29T11:30:00.144 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:30:06.950 DEBUG:teuthology.orchestra.run.smithi084:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:30:07.101 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:07.109+0000 7fb609354640 10 monclient: get_monmap_and_config 2024-03-29T11:30:07.101 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:07.109+0000 7fb609354640 10 monclient: build_initial_monmap 2024-03-29T11:30:07.101 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:07.109+0000 7fb609354640 10 monclient: monmap: 2024-03-29T11:30:07.101 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-29T11:30:07.101 INFO:teuthology.misc.health.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:30:07.102 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T11:30:07.110239+0000 2024-03-29T11:30:07.102 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T11:30:07.110239+0000 2024-03-29T11:30:07.102 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-29T11:30:07.102 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:30:07.102 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-29T11:30:07.102 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.noname-c 2024-03-29T11:30:07.102 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-b 2024-03-29T11:30:07.102 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:30:07.104 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:07.110+0000 7fb609354640 1 Processor -- start 2024-03-29T11:30:07.104 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:07.112+0000 7fb609354640 1 -- start start 2024-03-29T11:30:07.104 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:07.112+0000 7fb609354640 10 monclient: init 2024-03-29T11:30:07.105 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:07.113+0000 7fb609354640 10 monclient: _reopen_session rank -1 2024-03-29T11:30:07.105 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:07.113+0000 7fb609354640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:30:07.108 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:07.113+0000 7fb609354640 10 monclient(hunting): picked mon.noname-a con 0x7fb6041457f0 addr v1:172.21.15.84:6789/0 2024-03-29T11:30:07.111 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:07.116+0000 7fb609354640 10 monclient(hunting): picked mon.noname-c con 0x7fb60413f650 addr v1:172.21.15.179:6789/0 2024-03-29T11:30:07.111 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:07.117+0000 7fb609354640 10 monclient(hunting): picked mon.noname-b con 0x7fb6040a9d70 addr v1:172.21.15.84:6790/0 2024-03-29T11:30:07.112 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:07.117+0000 7fb609354640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb6040aaef0 con 0x7fb6041457f0 2024-03-29T11:30:07.112 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:07.117+0000 7fb609354640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb604145bd0 con 0x7fb60413f650 2024-03-29T11:30:07.112 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:07.117+0000 7fb609354640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb604145eb0 con 0x7fb6040a9d70 2024-03-29T11:30:07.112 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:07.117+0000 7fb609354640 10 monclient(hunting): _renew_subs 2024-03-29T11:30:07.112 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:07.117+0000 7fb609354640 10 monclient(hunting): authenticate will time out at 1468.411993s 2024-03-29T11:30:07.112 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:07.117+0000 7fb602ffd640 1 --1- >> v1:172.21.15.84:6789/0 conn(0x7fb6041457f0 0x7fb60413ef40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.84:6789/0 says I am v1:172.21.15.84:46283/0 (socket says 172.21.15.84:46283) 2024-03-29T11:30:07.112 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:07.117+0000 7fb602ffd640 1 -- 172.21.15.84:0/3931136347 learned_addr learned my addr 172.21.15.84:0/3931136347 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-29T11:30:07.112 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:07.118+0000 7fb601ffb640 1 -- 172.21.15.84:0/3931136347 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 545105856 0 0) 0x7fb604145eb0 con 0x7fb6040a9d70 2024-03-29T11:30:07.112 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:07.118+0000 7fb601ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:30:07.112 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:07.118+0000 7fb601ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:30:07.112 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:07.118+0000 7fb601ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:30:07.112 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:07.118+0000 7fb601ffb640 10 monclient(hunting): my global_id is 4673 2024-03-29T11:30:07.112 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:07.118+0000 7fb601ffb640 1 -- 172.21.15.84:0/3931136347 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb5ec003520 con 0x7fb6040a9d70 2024-03-29T11:30:07.112 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:07.118+0000 7fb601ffb640 1 -- 172.21.15.84:0/3931136347 <== mon.0 v1:172.21.15.84:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3635109782 0 0) 0x7fb6040aaef0 con 0x7fb6041457f0 2024-03-29T11:30:07.112 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:07.118+0000 7fb601ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:30:07.113 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:07.118+0000 7fb601ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:30:07.113 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:07.118+0000 7fb601ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:30:07.113 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:07.118+0000 7fb601ffb640 10 monclient(hunting): my global_id is 4596 2024-03-29T11:30:07.113 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:07.118+0000 7fb601ffb640 1 -- 172.21.15.84:0/3931136347 --> v1:172.21.15.84:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb604145eb0 con 0x7fb6041457f0 2024-03-29T11:30:07.114 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:07.120+0000 7fb601ffb640 1 -- 172.21.15.84:0/3931136347 <== mon.2 v1:172.21.15.84:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3973012154 0 0) 0x7fb5ec003520 con 0x7fb6040a9d70 2024-03-29T11:30:07.114 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:07.120+0000 7fb601ffb640 1 -- 172.21.15.84:0/3931136347 >> v1:172.21.15.179:6789/0 conn(0x7fb60413f650 legacy=0x7fb6040b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:30:07.114 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:07.120+0000 7fb601ffb640 1 -- 172.21.15.84:0/3931136347 >> v1:172.21.15.84:6789/0 conn(0x7fb6041457f0 legacy=0x7fb60413ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:30:07.115 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:07.120+0000 7fb601ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:30:07.115 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:07.120+0000 7fb601ffb640 1 monclient: found mon.noname-b 2024-03-29T11:30:07.115 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:07.120+0000 7fb601ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:30:07.115 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:07.120+0000 7fb601ffb640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.84:6790/0 2024-03-29T11:30:07.115 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:07.120+0000 7fb601ffb640 1 -- 172.21.15.84:0/3931136347 --> v1:172.21.15.84:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb604146340 con 0x7fb6040a9d70 2024-03-29T11:30:07.115 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:07.120+0000 7fb601ffb640 10 monclient: _finish_auth 0 2024-03-29T11:30:07.115 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:07.120+0000 7fb601ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:30:07.115 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:07.120+0000 7fb601ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:30:07.115 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:07.121+0000 7fb601ffb640 1 -- 172.21.15.84:0/3931136347 <== mon.2 v1:172.21.15.84:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4236868694 0 0) 0x7fb5fc001a10 con 0x7fb6040a9d70 2024-03-29T11:30:07.115 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:07.121+0000 7fb601ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:30:07.115 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:07.121+0000 7fb601ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:30:07.115 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:07.121+0000 7fb601ffb640 10 monclient: dump: 2024-03-29T11:30:07.115 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T11:30:07.115 INFO:teuthology.misc.health.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:30:07.116 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T11:18:53.864025+0000 2024-03-29T11:30:07.116 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T11:18:53.864025+0000 2024-03-29T11:30:07.116 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T11:30:07.116 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:30:07.116 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T11:30:07.116 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T11:30:07.116 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T11:30:07.116 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:30:07.116 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:07.121+0000 7fb609354640 5 monclient: authenticate success, global_id 4673 2024-03-29T11:30:07.116 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:07.121+0000 7fb609354640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:30:07.117 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:07.125+0000 7fb601ffb640 1 -- 172.21.15.84:0/3931136347 <== mon.2 v1:172.21.15.84:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb5fc001b70 con 0x7fb6040a9d70 2024-03-29T11:30:07.117 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:07.125+0000 7fb601ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:30:07.117 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:07.125+0000 7fb601ffb640 1 -- 172.21.15.84:0/3931136347 <== mon.2 v1:172.21.15.84:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4236868694 0 0) 0x7fb5fc001f80 con 0x7fb6040a9d70 2024-03-29T11:30:07.117 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:07.125+0000 7fb601ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:30:07.118 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:07.125+0000 7fb601ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:30:07.118 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:07.125+0000 7fb601ffb640 10 monclient: dump: 2024-03-29T11:30:07.118 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T11:30:07.118 INFO:teuthology.misc.health.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:30:07.118 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T11:18:53.864025+0000 2024-03-29T11:30:07.118 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T11:18:53.864025+0000 2024-03-29T11:30:07.118 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T11:30:07.118 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:30:07.118 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T11:30:07.118 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T11:30:07.118 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T11:30:07.118 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:30:07.119 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:07.125+0000 7fb609354640 10 monclient: get_monmap_and_config success 2024-03-29T11:30:07.119 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:07.125+0000 7fb609354640 4 set_mon_vals no callback set 2024-03-29T11:30:07.120 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:07.126+0000 7fb609354640 10 monclient: shutdown 2024-03-29T11:30:07.120 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:07.126+0000 7fb609354640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:30:07.120 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:07.126+0000 7fb609354640 1 -- 172.21.15.84:0/3931136347 >> v1:172.21.15.84:6790/0 conn(0x7fb6040a9d70 legacy=0x7fb60414d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:30:07.120 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:07.126+0000 7fb609354640 1 -- 172.21.15.84:0/3931136347 shutdown_connections 2024-03-29T11:30:07.120 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:07.126+0000 7fb609354640 1 -- 172.21.15.84:0/3931136347 >> 172.21.15.84:0/3931136347 conn(0x7fb604066a00 msgr2=0x7fb604066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:30:07.120 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:07.126+0000 7fb609354640 1 -- 172.21.15.84:0/3931136347 shutdown_connections 2024-03-29T11:30:07.120 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:07.126+0000 7fb609354640 1 -- 172.21.15.84:0/3931136347 wait complete. 2024-03-29T11:30:07.120 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:07.128+0000 7fb609354640 10 monclient: build_initial_monmap 2024-03-29T11:30:07.121 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:07.128+0000 7fb609354640 10 monclient: monmap: 2024-03-29T11:30:07.121 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-29T11:30:07.121 INFO:teuthology.misc.health.smithi084.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:30:07.121 INFO:teuthology.misc.health.smithi084.stderr:last_changed 0.000000 2024-03-29T11:30:07.121 INFO:teuthology.misc.health.smithi084.stderr:created 0.000000 2024-03-29T11:30:07.121 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-29T11:30:07.121 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:30:07.121 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-29T11:30:07.121 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.noname-b 2024-03-29T11:30:07.121 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-c 2024-03-29T11:30:07.121 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:30:07.125 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:07.128+0000 7fb609354640 1 librados: starting msgr at 2024-03-29T11:30:07.126 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:07.128+0000 7fb609354640 1 librados: starting objecter 2024-03-29T11:30:07.126 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:07.129+0000 7fb609354640 1 Processor -- start 2024-03-29T11:30:07.126 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:07.129+0000 7fb609354640 1 -- start start 2024-03-29T11:30:07.126 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:07.129+0000 7fb609354640 1 librados: setting wanted keys 2024-03-29T11:30:07.126 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:07.129+0000 7fb609354640 1 librados: calling monclient init 2024-03-29T11:30:07.126 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:07.129+0000 7fb609354640 10 monclient: init 2024-03-29T11:30:07.126 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:07.130+0000 7fb609354640 10 monclient: _reopen_session rank -1 2024-03-29T11:30:07.126 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:07.130+0000 7fb609354640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:30:07.126 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:07.130+0000 7fb609354640 10 monclient(hunting): picked mon.noname-c con 0x7fb6040a9d70 addr v1:172.21.15.84:6790/0 2024-03-29T11:30:07.126 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:07.130+0000 7fb609354640 10 monclient(hunting): picked mon.noname-a con 0x7fb60413f650 addr v1:172.21.15.84:6789/0 2024-03-29T11:30:07.126 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:07.130+0000 7fb609354640 10 monclient(hunting): picked mon.noname-b con 0x7fb6041457f0 addr v1:172.21.15.179:6789/0 2024-03-29T11:30:07.126 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:07.130+0000 7fb609354640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb604142a00 con 0x7fb60413f650 2024-03-29T11:30:07.127 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:07.130+0000 7fb609354640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb6041447f0 con 0x7fb6041457f0 2024-03-29T11:30:07.127 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:07.130+0000 7fb609354640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb6040b1760 con 0x7fb6040a9d70 2024-03-29T11:30:07.127 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:07.130+0000 7fb609354640 10 monclient(hunting): _renew_subs 2024-03-29T11:30:07.127 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:07.130+0000 7fb609354640 10 monclient(hunting): authenticate will time out at 1468.424289s 2024-03-29T11:30:07.127 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:07.130+0000 7fb6027fc640 1 --1- >> v1:172.21.15.84:6789/0 conn(0x7fb60413f650 0x7fb60406de90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.84:6789/0 says I am v1:172.21.15.84:58418/0 (socket says 172.21.15.84:58418) 2024-03-29T11:30:07.127 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:07.130+0000 7fb6027fc640 1 -- 172.21.15.84:0/2244377376 learned_addr learned my addr 172.21.15.84:0/2244377376 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-29T11:30:07.127 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:07.130+0000 7fb5eb7fe640 10 client.?.objecter ms_handle_connect 0x7fb60413f650 2024-03-29T11:30:07.127 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:07.130+0000 7fb5eb7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:30:07.127 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:07.130+0000 7fb5eb7fe640 1 -- 172.21.15.84:0/2244377376 <== mon.0 v1:172.21.15.84:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1539978037 0 0) 0x7fb604142a00 con 0x7fb60413f650 2024-03-29T11:30:07.127 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:07.130+0000 7fb5eb7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:30:07.127 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:07.130+0000 7fb5eb7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:30:07.127 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:07.130+0000 7fb5eb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:30:07.127 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:07.130+0000 7fb5eb7fe640 10 monclient(hunting): my global_id is 4599 2024-03-29T11:30:07.127 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:07.130+0000 7fb5eb7fe640 1 -- 172.21.15.84:0/2244377376 --> v1:172.21.15.84:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb5f0004eb0 con 0x7fb60413f650 2024-03-29T11:30:07.128 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:07.131+0000 7fb5eb7fe640 10 client.?.objecter ms_handle_connect 0x7fb6040a9d70 2024-03-29T11:30:07.128 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:07.131+0000 7fb5eb7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:30:07.128 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:07.131+0000 7fb5eb7fe640 1 -- 172.21.15.84:0/2244377376 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1726459343 0 0) 0x7fb6040b1760 con 0x7fb6040a9d70 2024-03-29T11:30:07.128 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:07.131+0000 7fb5eb7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:30:07.128 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:07.131+0000 7fb5eb7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:30:07.128 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:07.131+0000 7fb5eb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:30:07.128 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:07.131+0000 7fb5eb7fe640 10 monclient(hunting): my global_id is 4676 2024-03-29T11:30:07.128 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:07.131+0000 7fb5eb7fe640 1 -- 172.21.15.84:0/2244377376 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb604142a00 con 0x7fb6040a9d70 2024-03-29T11:30:07.128 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:07.131+0000 7fb5eb7fe640 1 -- 172.21.15.84:0/2244377376 <== mon.0 v1:172.21.15.84:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 504789341 0 0) 0x7fb5f0004eb0 con 0x7fb60413f650 2024-03-29T11:30:07.128 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:07.131+0000 7fb5eb7fe640 1 -- 172.21.15.84:0/2244377376 --> v1:172.21.15.84:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb6040b1760 con 0x7fb60413f650 2024-03-29T11:30:07.128 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:07.131+0000 7fb5eb7fe640 1 -- 172.21.15.84:0/2244377376 <== mon.0 v1:172.21.15.84:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4236868694 0 0) 0x7fb5f8001a30 con 0x7fb60413f650 2024-03-29T11:30:07.128 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:07.131+0000 7fb5eb7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:30:07.128 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:07.131+0000 7fb5eb7fe640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:30:07.128 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:07.131+0000 7fb5eb7fe640 10 monclient(hunting): dump: 2024-03-29T11:30:07.128 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T11:30:07.129 INFO:teuthology.misc.health.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:30:07.129 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T11:18:53.864025+0000 2024-03-29T11:30:07.129 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T11:18:53.864025+0000 2024-03-29T11:30:07.129 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T11:30:07.129 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:30:07.129 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T11:30:07.129 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T11:30:07.129 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T11:30:07.129 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:30:07.129 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:07.131+0000 7fb5eb7fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:30:07.129 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:07.131+0000 7fb5eb7fe640 1 -- 172.21.15.84:0/2244377376 <== mon.2 v1:172.21.15.84:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1231204111 0 0) 0x7fb604142a00 con 0x7fb6040a9d70 2024-03-29T11:30:07.129 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:07.131+0000 7fb5eb7fe640 1 -- 172.21.15.84:0/2244377376 --> v1:172.21.15.84:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb5f0005990 con 0x7fb6040a9d70 2024-03-29T11:30:07.129 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:07.131+0000 7fb5eb7fe640 1 -- 172.21.15.84:0/2244377376 <== mon.2 v1:172.21.15.84:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4236868694 0 0) 0x7fb5f4001850 con 0x7fb6040a9d70 2024-03-29T11:30:07.129 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:07.131+0000 7fb5eb7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:30:07.129 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:07.131+0000 7fb5eb7fe640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:30:07.130 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:07.131+0000 7fb5eb7fe640 10 monclient(hunting): dump: 2024-03-29T11:30:07.130 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T11:30:07.130 INFO:teuthology.misc.health.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:30:07.130 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T11:18:53.864025+0000 2024-03-29T11:30:07.130 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T11:18:53.864025+0000 2024-03-29T11:30:07.130 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T11:30:07.130 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:30:07.130 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T11:30:07.130 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T11:30:07.130 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T11:30:07.130 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:30:07.130 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:07.131+0000 7fb5eb7fe640 1 -- 172.21.15.84:0/2244377376 <== mon.0 v1:172.21.15.84:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1416805546 0 0) 0x7fb6040b1760 con 0x7fb60413f650 2024-03-29T11:30:07.131 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:07.132+0000 7fb5eb7fe640 1 -- 172.21.15.84:0/2244377376 >> v1:172.21.15.84:6790/0 conn(0x7fb6040a9d70 legacy=0x7fb604140d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:30:07.131 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:07.132+0000 7fb5eb7fe640 1 -- 172.21.15.84:0/2244377376 >> v1:172.21.15.179:6789/0 conn(0x7fb6041457f0 legacy=0x7fb6041e6740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:30:07.131 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:07.132+0000 7fb5eb7fe640 10 monclient: _finish_hunting 0 2024-03-29T11:30:07.131 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:07.132+0000 7fb5eb7fe640 1 monclient: found mon.a 2024-03-29T11:30:07.131 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:07.132+0000 7fb5eb7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:30:07.131 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:07.132+0000 7fb5eb7fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.84:6789/0 2024-03-29T11:30:07.131 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:07.132+0000 7fb5eb7fe640 1 -- 172.21.15.84:0/2244377376 --> v1:172.21.15.84:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb60413fda0 con 0x7fb60413f650 2024-03-29T11:30:07.131 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:07.132+0000 7fb5eb7fe640 10 monclient: _finish_auth 0 2024-03-29T11:30:07.131 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:07.132+0000 7fb5eb7fe640 10 monclient: _check_auth_tickets 2024-03-29T11:30:07.131 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:07.132+0000 7fb5eb7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:30:07.131 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:07.132+0000 7fb609354640 5 monclient: authenticate success, global_id 4599 2024-03-29T11:30:07.131 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:07.132+0000 7fb609354640 10 monclient: _renew_subs 2024-03-29T11:30:07.131 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:07.132+0000 7fb609354640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.84:6789/0 2024-03-29T11:30:07.131 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:07.132+0000 7fb609354640 1 -- 172.21.15.84:0/2244377376 --> v1:172.21.15.84:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb604141440 con 0x7fb60413f650 2024-03-29T11:30:07.131 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:07.132+0000 7fb609354640 10 client.4599.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:30:07.132 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:07.132+0000 7fb609354640 10 monclient: _renew_subs 2024-03-29T11:30:07.132 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:07.132+0000 7fb609354640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.84:6789/0 2024-03-29T11:30:07.132 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:07.132+0000 7fb609354640 1 -- 172.21.15.84:0/2244377376 --> v1:172.21.15.84:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb60406e780 con 0x7fb60413f650 2024-03-29T11:30:07.132 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:07.132+0000 7fb609354640 1 librados: init done 2024-03-29T11:30:07.132 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:07.132+0000 7fb5eb7fe640 1 -- 172.21.15.84:0/2244377376 <== mon.0 v1:172.21.15.84:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb5f8001c70 con 0x7fb60413f650 2024-03-29T11:30:07.132 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:07.132+0000 7fb5eb7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:30:07.132 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:07.132+0000 7fb5eb7fe640 1 -- 172.21.15.84:0/2244377376 <== mon.0 v1:172.21.15.84:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4236868694 0 0) 0x7fb5f8001a30 con 0x7fb60413f650 2024-03-29T11:30:07.132 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:07.132+0000 7fb5eb7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:30:07.132 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:07.132+0000 7fb5eb7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:30:07.132 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:07.132+0000 7fb5eb7fe640 10 monclient: dump: 2024-03-29T11:30:07.132 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T11:30:07.132 INFO:teuthology.misc.health.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:30:07.132 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T11:18:53.864025+0000 2024-03-29T11:30:07.132 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T11:18:53.864025+0000 2024-03-29T11:30:07.133 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T11:30:07.133 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:30:07.133 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T11:30:07.133 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T11:30:07.133 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T11:30:07.133 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:30:07.133 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:07.132+0000 7fb6017fa640 4 set_mon_vals no callback set 2024-03-29T11:30:07.133 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:07.133+0000 7fb609354640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:30:07.133 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:07.133+0000 7fb609354640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:30:07.133 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:07.133+0000 7fb609354640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.84:6789/0 2024-03-29T11:30:07.133 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:07.133+0000 7fb609354640 1 -- 172.21.15.84:0/2244377376 --> v1:172.21.15.84:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb5e4005080 con 0x7fb60413f650 2024-03-29T11:30:07.134 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:07.134+0000 7fb5eb7fe640 1 -- 172.21.15.84:0/2244377376 <== mon.0 v1:172.21.15.84:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4033942909 0 0) 0x7fb5f80038c0 con 0x7fb60413f650 2024-03-29T11:30:07.134 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:07.135+0000 7fb5eb7fe640 1 -- 172.21.15.84:0/2244377376 <== mon.0 v1:172.21.15.84:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 156031568 0 0) 0x7fb5f80847d0 con 0x7fb60413f650 2024-03-29T11:30:07.134 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:07.135+0000 7fb5eb7fe640 10 client.4599.objecter ms_dispatch 0x7fb604000b90 osd_map(11..11 src has 1..11) v4 2024-03-29T11:30:07.134 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:07.135+0000 7fb5eb7fe640 3 client.4599.objecter handle_osd_map got epochs [11,11] > 0 2024-03-29T11:30:07.134 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:07.135+0000 7fb5eb7fe640 3 client.4599.objecter handle_osd_map decoding full epoch 11 2024-03-29T11:30:07.134 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:07.135+0000 7fb5eb7fe640 20 client.4599.objecter dump_active .. 0 homeless 2024-03-29T11:30:07.135 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:07.135+0000 7fb602ffd640 10 monclient: get_auth_request con 0x7fb5f006e400 auth_method 0 2024-03-29T11:30:07.135 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:07.135+0000 7fb5eb7fe640 10 client.4599.objecter ms_handle_connect 0x7fb5f006e400 2024-03-29T11:30:07.135 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:07.143+0000 7fb5eb7fe640 1 -- 172.21.15.84:0/2244377376 <== mon.0 v1:172.21.15.84:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fb5f8050020 con 0x7fb60413f650 2024-03-29T11:30:07.135 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:07.143+0000 7fb5eb7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:30:07.135 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:07.143+0000 7fb5eb7fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:30:07.482 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:07.490+0000 7fb609354640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:30:07.482 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:07.490+0000 7fb609354640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:30:07.482 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:07.490+0000 7fb609354640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.84:6789/0 2024-03-29T11:30:07.482 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:07.490+0000 7fb609354640 1 -- 172.21.15.84:0/2244377376 --> v1:172.21.15.84:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb5e4005330 con 0x7fb60413f650 2024-03-29T11:30:07.483 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:07.491+0000 7fb5eb7fe640 1 -- 172.21.15.84:0/2244377376 <== mon.0 v1:172.21.15.84:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 782149409) 0x7fb5f8052120 con 0x7fb60413f650 2024-03-29T11:30:07.483 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:07.491+0000 7fb5eb7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:30:07.483 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:07.491+0000 7fb5eb7fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:30:07.483 INFO:teuthology.misc.health.smithi084.stdout: 2024-03-29T11:30:07.483 INFO:teuthology.misc.health.smithi084.stdout:{"epoch":11,"fsid":"2c3e002f-9eb0-4860-853b-4ef1ef562213","created":"2024-03-29T11:19:32.029169+0000","modified":"2024-03-29T11:29:36.955202+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:29:36.955202+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"5c39e276-2034-4716-bbc0-57a634eefa92","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"cd580e99-109f-4bf1-88c0-a5987bc8b9ed","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"11e60a5a-051d-453f-a190-66387cb35592","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"2c329fbe-bbc4-4030-bf87-d711cad04679","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f4213d4a-bd36-4433-91ae-d7e2b02481d3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"f0cf44ea-6055-48af-805b-aed858102ff9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"d895eaa0-cc0b-4ee4-9b5a-2dfcd89a60b7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"9bc5a7bc-8f52-4db9-96d6-de5e68d52eaa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:30:07.487 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:07.494+0000 7fb609354640 10 librados: watch_flush enter 2024-03-29T11:30:07.487 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:07.494+0000 7fb609354640 10 librados: watch_flush exit 2024-03-29T11:30:07.487 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:07.494+0000 7fb609354640 20 client.4599.objecter shutdown clearing up homeless session... 2024-03-29T11:30:07.487 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:07.494+0000 7fb609354640 10 client.4599.objecter successfully canceled tick 2024-03-29T11:30:07.487 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:07.494+0000 7fb609354640 1 -- 172.21.15.84:0/2244377376 >> v1:172.21.15.179:6816/2981134980 conn(0x7fb5f006e400 legacy=0x7fb5f00708c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:30:07.487 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:07.494+0000 7fb609354640 10 monclient: shutdown 2024-03-29T11:30:07.487 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:07.494+0000 7fb609354640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:30:07.487 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:07.494+0000 7fb609354640 1 -- 172.21.15.84:0/2244377376 >> v1:172.21.15.84:6789/0 conn(0x7fb60413f650 legacy=0x7fb60406de90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:30:07.487 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:07.494+0000 7fb609354640 1 -- 172.21.15.84:0/2244377376 shutdown_connections 2024-03-29T11:30:07.487 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:07.494+0000 7fb609354640 1 -- 172.21.15.84:0/2244377376 >> 172.21.15.84:0/2244377376 conn(0x7fb604066a00 msgr2=0x7fb6040b4820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:30:07.488 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:07.494+0000 7fb609354640 1 -- 172.21.15.84:0/2244377376 shutdown_connections 2024-03-29T11:30:07.488 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:07.494+0000 7fb609354640 1 -- 172.21.15.84:0/2244377376 wait complete. 2024-03-29T11:30:07.488 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:07.495+0000 7fb609354640 1 librados: shutdown 2024-03-29T11:30:07.502 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:30:14.307 DEBUG:teuthology.orchestra.run.smithi084:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:30:14.467 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:14.473+0000 7f1f3dca1640 10 monclient: get_monmap_and_config 2024-03-29T11:30:14.467 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:14.473+0000 7f1f3dca1640 10 monclient: build_initial_monmap 2024-03-29T11:30:14.467 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:14.473+0000 7f1f3dca1640 10 monclient: monmap: 2024-03-29T11:30:14.467 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-29T11:30:14.467 INFO:teuthology.misc.health.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:30:14.467 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T11:30:14.475365+0000 2024-03-29T11:30:14.468 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T11:30:14.475365+0000 2024-03-29T11:30:14.468 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-29T11:30:14.468 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:30:14.468 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-29T11:30:14.468 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.noname-c 2024-03-29T11:30:14.468 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-b 2024-03-29T11:30:14.468 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:30:14.468 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:14.474+0000 7f1f3dca1640 1 Processor -- start 2024-03-29T11:30:14.469 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:14.475+0000 7f1f3dca1640 1 -- start start 2024-03-29T11:30:14.469 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:14.475+0000 7f1f3dca1640 10 monclient: init 2024-03-29T11:30:14.469 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:14.476+0000 7f1f3dca1640 10 monclient: _reopen_session rank -1 2024-03-29T11:30:14.469 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:14.476+0000 7f1f3dca1640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:30:14.469 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:14.476+0000 7f1f3dca1640 10 monclient(hunting): picked mon.noname-a con 0x7f1f380a9ac0 addr v1:172.21.15.84:6789/0 2024-03-29T11:30:14.469 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:14.476+0000 7f1f3dca1640 10 monclient(hunting): picked mon.noname-b con 0x7f1f3813ee90 addr v1:172.21.15.84:6790/0 2024-03-29T11:30:14.469 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:14.476+0000 7f1f3dca1640 10 monclient(hunting): picked mon.noname-c con 0x7f1f380af900 addr v1:172.21.15.179:6789/0 2024-03-29T11:30:14.469 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:14.476+0000 7f1f3dca1640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1f380afd10 con 0x7f1f380a9ac0 2024-03-29T11:30:14.469 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:14.476+0000 7f1f3dca1640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1f380af6f0 con 0x7f1f380af900 2024-03-29T11:30:14.470 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:14.476+0000 7f1f3dca1640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1f380aac80 con 0x7f1f3813ee90 2024-03-29T11:30:14.470 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:14.476+0000 7f1f3dca1640 10 monclient(hunting): _renew_subs 2024-03-29T11:30:14.470 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:14.476+0000 7f1f3dca1640 10 monclient(hunting): authenticate will time out at 1475.771270s 2024-03-29T11:30:14.471 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:14.476+0000 7f1f377fe640 1 --1- >> v1:172.21.15.84:6789/0 conn(0x7f1f380a9ac0 0x7f1f380adfe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.84:6789/0 says I am v1:172.21.15.84:42726/0 (socket says 172.21.15.84:42726) 2024-03-29T11:30:14.471 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:14.476+0000 7f1f377fe640 1 -- 172.21.15.84:0/529147641 learned_addr learned my addr 172.21.15.84:0/529147641 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-29T11:30:14.471 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:14.476+0000 7f1f367fc640 1 -- 172.21.15.84:0/529147641 <== mon.0 v1:172.21.15.84:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 107430427 0 0) 0x7f1f380afd10 con 0x7f1f380a9ac0 2024-03-29T11:30:14.471 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:14.476+0000 7f1f367fc640 10 monclient(hunting): _negotiate 2024-03-29T11:30:14.471 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:14.476+0000 7f1f367fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:30:14.471 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:14.476+0000 7f1f367fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:30:14.472 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:14.476+0000 7f1f367fc640 10 monclient(hunting): my global_id is 4602 2024-03-29T11:30:14.472 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:14.476+0000 7f1f367fc640 1 -- 172.21.15.84:0/529147641 --> v1:172.21.15.84:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1f20003520 con 0x7f1f380a9ac0 2024-03-29T11:30:14.472 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:14.477+0000 7f1f367fc640 1 -- 172.21.15.84:0/529147641 <== mon.0 v1:172.21.15.84:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2285575247 0 0) 0x7f1f20003520 con 0x7f1f380a9ac0 2024-03-29T11:30:14.472 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:14.477+0000 7f1f367fc640 1 -- 172.21.15.84:0/529147641 >> v1:172.21.15.84:6790/0 conn(0x7f1f3813ee90 legacy=0x7f1f3813b5f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:30:14.473 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:14.480+0000 7f1f367fc640 1 -- 172.21.15.84:0/529147641 >> v1:172.21.15.179:6789/0 conn(0x7f1f380af900 legacy=0x7f1f38149b00 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:30:14.473 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:14.480+0000 7f1f367fc640 10 monclient: _finish_hunting 0 2024-03-29T11:30:14.474 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:14.480+0000 7f1f367fc640 1 monclient: found mon.noname-a 2024-03-29T11:30:14.474 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:14.480+0000 7f1f367fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:30:14.474 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:14.480+0000 7f1f367fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.84:6789/0 2024-03-29T11:30:14.477 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:14.480+0000 7f1f367fc640 1 -- 172.21.15.84:0/529147641 --> v1:172.21.15.84:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f380aaf60 con 0x7f1f380a9ac0 2024-03-29T11:30:14.477 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:14.480+0000 7f1f367fc640 10 monclient: _finish_auth 0 2024-03-29T11:30:14.477 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:14.480+0000 7f1f367fc640 10 monclient: _check_auth_tickets 2024-03-29T11:30:14.477 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:14.480+0000 7f1f367fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:30:14.478 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:14.480+0000 7f1f367fc640 1 -- 172.21.15.84:0/529147641 <== mon.0 v1:172.21.15.84:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4236868694 0 0) 0x7f1f28001690 con 0x7f1f380a9ac0 2024-03-29T11:30:14.478 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:14.480+0000 7f1f367fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:30:14.478 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:14.480+0000 7f1f367fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:30:14.478 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:14.480+0000 7f1f367fc640 10 monclient: dump: 2024-03-29T11:30:14.478 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T11:30:14.478 INFO:teuthology.misc.health.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:30:14.478 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T11:18:53.864025+0000 2024-03-29T11:30:14.478 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T11:18:53.864025+0000 2024-03-29T11:30:14.478 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T11:30:14.478 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:30:14.478 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T11:30:14.478 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T11:30:14.478 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T11:30:14.479 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:30:14.479 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:14.480+0000 7f1f3dca1640 5 monclient: authenticate success, global_id 4602 2024-03-29T11:30:14.479 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:14.480+0000 7f1f3dca1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:30:14.479 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:14.480+0000 7f1f377fe640 1 -- 172.21.15.84:0/529147641 >> v1:172.21.15.84:6789/0 conn(0x7f1f380a9ac0 legacy=0x7f1f380adfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-03-29T11:30:14.479 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:14.480+0000 7f1f377fe640 1 -- 172.21.15.84:0/529147641 >> v1:172.21.15.84:6789/0 conn(0x7f1f380a9ac0 legacy=0x7f1f380adfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-29T11:30:14.479 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:14.480+0000 7f1f377fe640 1 --1- 172.21.15.84:0/529147641 >> v1:172.21.15.84:6789/0 conn(0x7f1f380a9ac0 0x7f1f380adfe0 :-1 s=OPENED pgs=1061589 cs=1 l=1).handle_message read tag failed 2024-03-29T11:30:14.479 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:14.480+0000 7f1f377fe640 1 --1- 172.21.15.84:0/529147641 >> v1:172.21.15.84:6789/0 conn(0x7f1f380a9ac0 0x7f1f380adfe0 :-1 s=OPENED pgs=1061589 cs=1 l=1).fault on lossy channel, failing 2024-03-29T11:30:14.479 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:14.480+0000 7f1f367fc640 10 monclient: ms_handle_reset current mon v1:172.21.15.84:6789/0 2024-03-29T11:30:14.479 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:14.480+0000 7f1f367fc640 10 monclient: _reopen_session rank -1 2024-03-29T11:30:14.479 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:14.480+0000 7f1f367fc640 1 -- 172.21.15.84:0/529147641 >> v1:172.21.15.84:6789/0 conn(0x7f1f380a9ac0 legacy=0x7f1f380adfe0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-29T11:30:14.479 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:14.480+0000 7f1f367fc640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:30:14.479 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:14.480+0000 7f1f367fc640 10 monclient(hunting): picked mon.a con 0x7f1f20003af0 addr v1:172.21.15.84:6789/0 2024-03-29T11:30:14.479 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:14.480+0000 7f1f367fc640 10 monclient(hunting): picked mon.b con 0x7f1f200079e0 addr v1:172.21.15.179:6789/0 2024-03-29T11:30:14.479 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:14.480+0000 7f1f367fc640 10 monclient(hunting): picked mon.c con 0x7f1f3813ee90 addr v1:172.21.15.84:6790/0 2024-03-29T11:30:14.479 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:14.480+0000 7f1f367fc640 1 -- 172.21.15.84:0/529147641 --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f1f20003520 con 0x7f1f20003af0 2024-03-29T11:30:14.480 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:14.480+0000 7f1f367fc640 1 -- 172.21.15.84:0/529147641 --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f1f380af6f0 con 0x7f1f200079e0 2024-03-29T11:30:14.480 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:14.480+0000 7f1f367fc640 1 -- 172.21.15.84:0/529147641 --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f1f380aac80 con 0x7f1f3813ee90 2024-03-29T11:30:14.480 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:14.480+0000 7f1f367fc640 10 monclient(hunting): _renew_subs 2024-03-29T11:30:14.480 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:14.480+0000 7f1f367fc640 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.84:6789/0 2024-03-29T11:30:14.480 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:14.482+0000 7f1f367fc640 1 -- 172.21.15.84:0/529147641 <== mon.0 v1:172.21.15.84:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 150227111 0 0) 0x7f1f20003520 con 0x7f1f20003af0 2024-03-29T11:30:14.480 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:14.482+0000 7f1f367fc640 10 monclient(hunting): _negotiate 2024-03-29T11:30:14.480 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:14.482+0000 7f1f367fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:30:14.480 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:14.482+0000 7f1f367fc640 10 monclient(hunting): _init_auth already have auth, reseting 2024-03-29T11:30:14.480 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:14.482+0000 7f1f367fc640 1 -- 172.21.15.84:0/529147641 --> v1:172.21.15.84:6789/0 -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f1f200137d0 con 0x7f1f20003af0 2024-03-29T11:30:14.480 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:14.482+0000 7f1f367fc640 1 -- 172.21.15.84:0/529147641 <== mon.0 v1:172.21.15.84:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 233+0+0 (unknown 4283115150 0 0) 0x7f1f200137d0 con 0x7f1f20003af0 2024-03-29T11:30:14.480 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:14.482+0000 7f1f367fc640 1 -- 172.21.15.84:0/529147641 >> v1:172.21.15.84:6790/0 conn(0x7f1f3813ee90 legacy=0x7f1f200120b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:30:14.482 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:14.486+0000 7f1f367fc640 1 -- 172.21.15.84:0/529147641 >> v1:172.21.15.179:6789/0 conn(0x7f1f200079e0 legacy=0x7f1f20007e60 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-29T11:30:14.482 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:14.486+0000 7f1f367fc640 10 monclient: _finish_hunting 0 2024-03-29T11:30:14.482 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:14.486+0000 7f1f367fc640 1 monclient: found mon.a 2024-03-29T11:30:14.482 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:14.486+0000 7f1f367fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:30:14.482 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:14.486+0000 7f1f367fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.84:6789/0 2024-03-29T11:30:14.482 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:14.486+0000 7f1f367fc640 1 -- 172.21.15.84:0/529147641 --> v1:172.21.15.84:6789/0 -- mon_subscribe({config=0+,monmap=2+}) v3 -- 0x7f1f380afd10 con 0x7f1f20003af0 2024-03-29T11:30:14.482 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:14.486+0000 7f1f367fc640 10 monclient: _finish_auth 0 2024-03-29T11:30:14.482 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:14.486+0000 7f1f367fc640 10 monclient: _check_auth_tickets 2024-03-29T11:30:14.483 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:14.486+0000 7f1f367fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:30:14.483 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:14.486+0000 7f1f367fc640 1 -- 172.21.15.84:0/529147641 <== mon.0 v1:172.21.15.84:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4236868694 0 0) 0x7f1f28004a10 con 0x7f1f20003af0 2024-03-29T11:30:14.483 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:14.486+0000 7f1f367fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:30:14.483 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:14.486+0000 7f1f367fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:30:14.483 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:14.486+0000 7f1f367fc640 10 monclient: dump: 2024-03-29T11:30:14.483 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T11:30:14.483 INFO:teuthology.misc.health.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:30:14.483 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T11:18:53.864025+0000 2024-03-29T11:30:14.483 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T11:18:53.864025+0000 2024-03-29T11:30:14.483 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T11:30:14.483 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:30:14.483 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T11:30:14.483 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T11:30:14.483 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T11:30:14.484 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:30:14.484 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:14.486+0000 7f1f3dca1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:30:14.484 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:14.488+0000 7f1f367fc640 1 -- 172.21.15.84:0/529147641 <== mon.0 v1:172.21.15.84:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1f28004b70 con 0x7f1f20003af0 2024-03-29T11:30:14.484 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:14.488+0000 7f1f367fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:30:14.484 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:14.491+0000 7f1f3dca1640 10 monclient: get_monmap_and_config success 2024-03-29T11:30:14.484 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:14.491+0000 7f1f3dca1640 4 set_mon_vals no callback set 2024-03-29T11:30:14.485 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:14.492+0000 7f1f3dca1640 10 monclient: shutdown 2024-03-29T11:30:14.485 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:14.492+0000 7f1f3dca1640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:30:14.485 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:14.492+0000 7f1f3dca1640 1 -- 172.21.15.84:0/529147641 >> v1:172.21.15.84:6789/0 conn(0x7f1f20003af0 legacy=0x7f1f20003ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:30:14.485 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:14.492+0000 7f1f3dca1640 1 -- 172.21.15.84:0/529147641 shutdown_connections 2024-03-29T11:30:14.485 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:14.492+0000 7f1f3dca1640 1 -- 172.21.15.84:0/529147641 >> 172.21.15.84:0/529147641 conn(0x7f1f380b35e0 msgr2=0x7f1f380b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:30:14.488 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:14.493+0000 7f1f3dca1640 1 -- 172.21.15.84:0/529147641 shutdown_connections 2024-03-29T11:30:14.493 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:14.497+0000 7f1f3dca1640 1 -- 172.21.15.84:0/529147641 wait complete. 2024-03-29T11:30:14.493 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:14.498+0000 7f1f3dca1640 10 monclient: build_initial_monmap 2024-03-29T11:30:14.493 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:14.498+0000 7f1f3dca1640 10 monclient: monmap: 2024-03-29T11:30:14.493 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-29T11:30:14.493 INFO:teuthology.misc.health.smithi084.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:30:14.493 INFO:teuthology.misc.health.smithi084.stderr:last_changed 0.000000 2024-03-29T11:30:14.493 INFO:teuthology.misc.health.smithi084.stderr:created 0.000000 2024-03-29T11:30:14.494 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-29T11:30:14.494 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:30:14.494 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-29T11:30:14.494 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.noname-b 2024-03-29T11:30:14.494 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-c 2024-03-29T11:30:14.494 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:30:14.494 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:14.498+0000 7f1f3dca1640 1 librados: starting msgr at 2024-03-29T11:30:14.494 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:14.498+0000 7f1f3dca1640 1 librados: starting objecter 2024-03-29T11:30:14.494 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:14.498+0000 7f1f3dca1640 1 Processor -- start 2024-03-29T11:30:14.494 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:14.499+0000 7f1f3dca1640 1 -- start start 2024-03-29T11:30:14.494 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:14.499+0000 7f1f3dca1640 1 librados: setting wanted keys 2024-03-29T11:30:14.494 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:14.499+0000 7f1f3dca1640 1 librados: calling monclient init 2024-03-29T11:30:14.494 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:14.499+0000 7f1f3dca1640 10 monclient: init 2024-03-29T11:30:14.494 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:14.499+0000 7f1f3dca1640 10 monclient: _reopen_session rank -1 2024-03-29T11:30:14.495 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:14.499+0000 7f1f3dca1640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:30:14.495 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:14.499+0000 7f1f3dca1640 10 monclient(hunting): picked mon.noname-a con 0x7f1f20003af0 addr v1:172.21.15.84:6789/0 2024-03-29T11:30:14.495 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:14.499+0000 7f1f3dca1640 10 monclient(hunting): picked mon.noname-c con 0x7f1f200079e0 addr v1:172.21.15.84:6790/0 2024-03-29T11:30:14.495 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:14.499+0000 7f1f3dca1640 10 monclient(hunting): picked mon.noname-b con 0x7f1f3813ee90 addr v1:172.21.15.179:6789/0 2024-03-29T11:30:14.495 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:14.499+0000 7f1f3dca1640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1f381414b0 con 0x7f1f20003af0 2024-03-29T11:30:14.495 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:14.499+0000 7f1f3dca1640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1f3806f290 con 0x7f1f3813ee90 2024-03-29T11:30:14.495 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:14.499+0000 7f1f3dca1640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1f38142150 con 0x7f1f200079e0 2024-03-29T11:30:14.495 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:14.499+0000 7f1f3dca1640 10 monclient(hunting): _renew_subs 2024-03-29T11:30:14.495 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:14.499+0000 7f1f3dca1640 10 monclient(hunting): authenticate will time out at 1475.794896s 2024-03-29T11:30:14.498 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:14.503+0000 7f1f377fe640 1 --1- >> v1:172.21.15.84:6789/0 conn(0x7f1f20003af0 0x7f1f3813cf60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.84:6789/0 says I am v1:172.21.15.84:54482/0 (socket says 172.21.15.84:54482) 2024-03-29T11:30:14.498 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:14.503+0000 7f1f377fe640 1 -- 172.21.15.84:0/2551365760 learned_addr learned my addr 172.21.15.84:0/2551365760 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-29T11:30:14.498 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:14.503+0000 7f1f1ffff640 10 client.?.objecter ms_handle_connect 0x7f1f3813ee90 2024-03-29T11:30:14.498 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:14.503+0000 7f1f1ffff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:30:14.498 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:14.503+0000 7f1f1ffff640 10 client.?.objecter ms_handle_connect 0x7f1f20003af0 2024-03-29T11:30:14.498 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:14.503+0000 7f1f1ffff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:30:14.498 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:14.503+0000 7f1f1ffff640 1 -- 172.21.15.84:0/2551365760 <== mon.1 v1:172.21.15.179:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3724698841 0 0) 0x7f1f3806f290 con 0x7f1f3813ee90 2024-03-29T11:30:14.498 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:14.504+0000 7f1f1ffff640 10 monclient(hunting): _negotiate 2024-03-29T11:30:14.498 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:14.504+0000 7f1f1ffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:30:14.499 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:14.504+0000 7f1f1ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:30:14.499 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:14.504+0000 7f1f1ffff640 10 monclient(hunting): my global_id is 4546 2024-03-29T11:30:14.499 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:14.504+0000 7f1f1ffff640 1 -- 172.21.15.84:0/2551365760 --> v1:172.21.15.179:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1f24004eb0 con 0x7f1f3813ee90 2024-03-29T11:30:14.499 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:14.504+0000 7f1f1ffff640 1 -- 172.21.15.84:0/2551365760 <== mon.1 v1:172.21.15.179:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2172537154 0 0) 0x7f1f24004eb0 con 0x7f1f3813ee90 2024-03-29T11:30:14.499 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:14.504+0000 7f1f1ffff640 1 -- 172.21.15.84:0/2551365760 --> v1:172.21.15.179:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1f3806f290 con 0x7f1f3813ee90 2024-03-29T11:30:14.499 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:14.504+0000 7f1f1ffff640 1 -- 172.21.15.84:0/2551365760 <== mon.1 v1:172.21.15.179:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4236868694 0 0) 0x7f1f30001a50 con 0x7f1f3813ee90 2024-03-29T11:30:14.499 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:14.504+0000 7f1f1ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:30:14.499 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:14.504+0000 7f1f1ffff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:30:14.499 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:14.504+0000 7f1f1ffff640 10 monclient(hunting): dump: 2024-03-29T11:30:14.499 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T11:30:14.499 INFO:teuthology.misc.health.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:30:14.499 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T11:18:53.864025+0000 2024-03-29T11:30:14.499 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T11:18:53.864025+0000 2024-03-29T11:30:14.499 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T11:30:14.499 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:30:14.500 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T11:30:14.500 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T11:30:14.500 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T11:30:14.500 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:30:14.500 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:14.504+0000 7f1f1ffff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:30:14.500 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:14.504+0000 7f1f1ffff640 1 -- 172.21.15.84:0/2551365760 <== mon.0 v1:172.21.15.84:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1576287727 0 0) 0x7f1f381414b0 con 0x7f1f20003af0 2024-03-29T11:30:14.500 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:14.504+0000 7f1f1ffff640 10 monclient(hunting): _negotiate 2024-03-29T11:30:14.500 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:14.504+0000 7f1f1ffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:30:14.500 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:14.504+0000 7f1f1ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:30:14.500 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:14.504+0000 7f1f1ffff640 10 monclient(hunting): my global_id is 4605 2024-03-29T11:30:14.500 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:14.504+0000 7f1f1ffff640 1 -- 172.21.15.84:0/2551365760 --> v1:172.21.15.84:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1f24004eb0 con 0x7f1f20003af0 2024-03-29T11:30:14.500 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:14.504+0000 7f1f1ffff640 1 -- 172.21.15.84:0/2551365760 <== mon.1 v1:172.21.15.179:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 219014121 0 0) 0x7f1f3806f290 con 0x7f1f3813ee90 2024-03-29T11:30:14.500 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:14.504+0000 7f1f1ffff640 1 -- 172.21.15.84:0/2551365760 >> v1:172.21.15.84:6790/0 conn(0x7f1f200079e0 legacy=0x7f1f38140ba0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:30:14.501 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:14.506+0000 7f1f1ffff640 1 -- 172.21.15.84:0/2551365760 >> v1:172.21.15.84:6789/0 conn(0x7f1f20003af0 legacy=0x7f1f3813cf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:30:14.502 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:14.506+0000 7f1f1ffff640 10 monclient: _finish_hunting 0 2024-03-29T11:30:14.502 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:14.506+0000 7f1f1ffff640 1 monclient: found mon.b 2024-03-29T11:30:14.502 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:14.506+0000 7f1f1ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:30:14.502 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:14.506+0000 7f1f1ffff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-29T11:30:14.502 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:14.506+0000 7f1f1ffff640 1 -- 172.21.15.84:0/2551365760 --> v1:172.21.15.179:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f38140510 con 0x7f1f3813ee90 2024-03-29T11:30:14.502 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:14.506+0000 7f1f1ffff640 10 monclient: _finish_auth 0 2024-03-29T11:30:14.503 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:14.506+0000 7f1f1ffff640 10 monclient: _check_auth_tickets 2024-03-29T11:30:14.503 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:14.506+0000 7f1f1ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:30:14.503 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:14.507+0000 7f1f3dca1640 5 monclient: authenticate success, global_id 4546 2024-03-29T11:30:14.503 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:14.507+0000 7f1f3dca1640 10 monclient: _renew_subs 2024-03-29T11:30:14.503 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:14.507+0000 7f1f3dca1640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-29T11:30:14.503 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:14.507+0000 7f1f3dca1640 1 -- 172.21.15.84:0/2551365760 --> v1:172.21.15.179:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1f381407c0 con 0x7f1f3813ee90 2024-03-29T11:30:14.503 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:14.507+0000 7f1f3dca1640 10 client.4546.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:30:14.503 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:14.507+0000 7f1f3dca1640 10 monclient: _renew_subs 2024-03-29T11:30:14.503 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:14.507+0000 7f1f3dca1640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-29T11:30:14.503 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:14.507+0000 7f1f3dca1640 1 -- 172.21.15.84:0/2551365760 --> v1:172.21.15.179:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f1f3813f7a0 con 0x7f1f3813ee90 2024-03-29T11:30:14.503 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:14.507+0000 7f1f3dca1640 1 librados: init done 2024-03-29T11:30:14.503 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:14.507+0000 7f1f1ffff640 1 -- 172.21.15.84:0/2551365760 <== mon.1 v1:172.21.15.179:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1f30001c20 con 0x7f1f3813ee90 2024-03-29T11:30:14.503 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:14.507+0000 7f1f1ffff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:30:14.503 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:14.507+0000 7f1f1ffff640 1 -- 172.21.15.84:0/2551365760 <== mon.1 v1:172.21.15.179:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4236868694 0 0) 0x7f1f30002160 con 0x7f1f3813ee90 2024-03-29T11:30:14.504 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:14.507+0000 7f1f35ffb640 4 set_mon_vals no callback set 2024-03-29T11:30:14.504 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:14.508+0000 7f1f1ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:30:14.504 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:14.508+0000 7f1f1ffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:30:14.504 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:14.508+0000 7f1f1ffff640 10 monclient: dump: 2024-03-29T11:30:14.504 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T11:30:14.504 INFO:teuthology.misc.health.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:30:14.504 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T11:18:53.864025+0000 2024-03-29T11:30:14.504 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T11:18:53.864025+0000 2024-03-29T11:30:14.504 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T11:30:14.504 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:30:14.504 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T11:30:14.504 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T11:30:14.504 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T11:30:14.504 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:30:14.504 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:14.508+0000 7f1f3dca1640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:30:14.505 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:14.508+0000 7f1f3dca1640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:30:14.505 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:14.508+0000 7f1f3dca1640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-29T11:30:14.505 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:14.509+0000 7f1f1ffff640 1 -- 172.21.15.84:0/2551365760 <== mon.1 v1:172.21.15.179:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4033942909 0 0) 0x7f1f300198a0 con 0x7f1f3813ee90 2024-03-29T11:30:14.505 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:14.510+0000 7f1f1ffff640 1 -- 172.21.15.84:0/2551365760 <== mon.1 v1:172.21.15.179:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 156031568 0 0) 0x7f1f300837f0 con 0x7f1f3813ee90 2024-03-29T11:30:14.506 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:14.510+0000 7f1f1ffff640 10 client.4546.objecter ms_dispatch 0x7f1f38000b90 osd_map(11..11 src has 1..11) v4 2024-03-29T11:30:14.506 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:14.510+0000 7f1f1ffff640 3 client.4546.objecter handle_osd_map got epochs [11,11] > 0 2024-03-29T11:30:14.506 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:14.510+0000 7f1f1ffff640 3 client.4546.objecter handle_osd_map decoding full epoch 11 2024-03-29T11:30:14.506 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:14.510+0000 7f1f1ffff640 20 client.4546.objecter dump_active .. 0 homeless 2024-03-29T11:30:14.506 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:14.508+0000 7f1f3dca1640 1 -- 172.21.15.84:0/2551365760 --> v1:172.21.15.179:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1f18005080 con 0x7f1f3813ee90 2024-03-29T11:30:14.506 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:14.510+0000 7f1f377fe640 10 monclient: get_auth_request con 0x7f1f2406dd60 auth_method 0 2024-03-29T11:30:14.506 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:14.511+0000 7f1f1ffff640 10 client.4546.objecter ms_handle_connect 0x7f1f2406dd60 2024-03-29T11:30:14.511 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:14.517+0000 7f1f1ffff640 1 -- 172.21.15.84:0/2551365760 <== mon.1 v1:172.21.15.179:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f1f3004ef00 con 0x7f1f3813ee90 2024-03-29T11:30:14.511 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:14.518+0000 7f1f1ffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:30:14.511 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:14.518+0000 7f1f1ffff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:30:14.867 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:14.873+0000 7f1f3dca1640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:30:14.867 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:14.873+0000 7f1f3dca1640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:30:14.867 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:14.873+0000 7f1f3dca1640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-29T11:30:14.867 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:14.873+0000 7f1f3dca1640 1 -- 172.21.15.84:0/2551365760 --> v1:172.21.15.179:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1f18005330 con 0x7f1f3813ee90 2024-03-29T11:30:14.869 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:14.875+0000 7f1f1ffff640 1 -- 172.21.15.84:0/2551365760 <== mon.1 v1:172.21.15.179:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 782149409) 0x7f1f30051000 con 0x7f1f3813ee90 2024-03-29T11:30:14.869 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:14.876+0000 7f1f1ffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:30:14.869 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:14.876+0000 7f1f1ffff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:30:14.870 INFO:teuthology.misc.health.smithi084.stdout: 2024-03-29T11:30:14.870 INFO:teuthology.misc.health.smithi084.stdout:{"epoch":11,"fsid":"2c3e002f-9eb0-4860-853b-4ef1ef562213","created":"2024-03-29T11:19:32.029169+0000","modified":"2024-03-29T11:29:36.955202+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:29:36.955202+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"5c39e276-2034-4716-bbc0-57a634eefa92","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"cd580e99-109f-4bf1-88c0-a5987bc8b9ed","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"11e60a5a-051d-453f-a190-66387cb35592","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"2c329fbe-bbc4-4030-bf87-d711cad04679","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f4213d4a-bd36-4433-91ae-d7e2b02481d3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"f0cf44ea-6055-48af-805b-aed858102ff9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"d895eaa0-cc0b-4ee4-9b5a-2dfcd89a60b7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"9bc5a7bc-8f52-4db9-96d6-de5e68d52eaa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:30:14.874 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:14.880+0000 7f1f3dca1640 10 librados: watch_flush enter 2024-03-29T11:30:14.874 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:14.880+0000 7f1f3dca1640 10 librados: watch_flush exit 2024-03-29T11:30:14.874 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:14.880+0000 7f1f3dca1640 20 client.4546.objecter shutdown clearing up homeless session... 2024-03-29T11:30:14.875 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:14.880+0000 7f1f3dca1640 10 client.4546.objecter successfully canceled tick 2024-03-29T11:30:14.875 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:14.880+0000 7f1f3dca1640 1 -- 172.21.15.84:0/2551365760 >> v1:172.21.15.179:6816/2981134980 conn(0x7f1f2406dd60 legacy=0x7f1f24070220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:30:14.875 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:14.880+0000 7f1f3dca1640 10 monclient: shutdown 2024-03-29T11:30:14.875 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:14.880+0000 7f1f3dca1640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:30:14.875 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:14.880+0000 7f1f3dca1640 1 -- 172.21.15.84:0/2551365760 >> v1:172.21.15.179:6789/0 conn(0x7f1f3813ee90 legacy=0x7f1f38141a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:30:14.875 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:14.881+0000 7f1f3dca1640 1 -- 172.21.15.84:0/2551365760 shutdown_connections 2024-03-29T11:30:14.875 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:14.881+0000 7f1f3dca1640 1 -- 172.21.15.84:0/2551365760 >> 172.21.15.84:0/2551365760 conn(0x7f1f380b35e0 msgr2=0x7f1f3813aaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:30:14.875 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:14.881+0000 7f1f3dca1640 1 -- 172.21.15.84:0/2551365760 shutdown_connections 2024-03-29T11:30:14.875 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:14.881+0000 7f1f3dca1640 1 -- 172.21.15.84:0/2551365760 wait complete. 2024-03-29T11:30:14.875 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:14.882+0000 7f1f3dca1640 1 librados: shutdown 2024-03-29T11:30:14.898 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:30:21.706 DEBUG:teuthology.orchestra.run.smithi084:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:30:21.854 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:21.861+0000 7fa75c5e2640 10 monclient: get_monmap_and_config 2024-03-29T11:30:21.854 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:21.861+0000 7fa75c5e2640 10 monclient: build_initial_monmap 2024-03-29T11:30:21.854 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:21.861+0000 7fa75c5e2640 10 monclient: monmap: 2024-03-29T11:30:21.854 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-29T11:30:21.854 INFO:teuthology.misc.health.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:30:21.854 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T11:30:21.862382+0000 2024-03-29T11:30:21.854 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T11:30:21.862382+0000 2024-03-29T11:30:21.854 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-29T11:30:21.854 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:30:21.854 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-29T11:30:21.854 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.noname-c 2024-03-29T11:30:21.854 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-b 2024-03-29T11:30:21.855 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:30:21.855 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:21.862+0000 7fa75c5e2640 1 Processor -- start 2024-03-29T11:30:21.855 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:21.862+0000 7fa75c5e2640 1 -- start start 2024-03-29T11:30:21.855 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:21.862+0000 7fa75c5e2640 10 monclient: init 2024-03-29T11:30:21.858 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:21.863+0000 7fa75c5e2640 10 monclient: _reopen_session rank -1 2024-03-29T11:30:21.858 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:21.863+0000 7fa75c5e2640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:30:21.858 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:21.863+0000 7fa75c5e2640 10 monclient(hunting): picked mon.noname-c con 0x7fa7541457f0 addr v1:172.21.15.179:6789/0 2024-03-29T11:30:21.858 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:21.863+0000 7fa75c5e2640 10 monclient(hunting): picked mon.noname-a con 0x7fa75413f650 addr v1:172.21.15.84:6789/0 2024-03-29T11:30:21.858 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:21.863+0000 7fa75c5e2640 10 monclient(hunting): picked mon.noname-b con 0x7fa7540a9d70 addr v1:172.21.15.84:6790/0 2024-03-29T11:30:21.858 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:21.863+0000 7fa75c5e2640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa7540aaef0 con 0x7fa75413f650 2024-03-29T11:30:21.858 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:21.863+0000 7fa75c5e2640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa754145bd0 con 0x7fa7541457f0 2024-03-29T11:30:21.858 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:21.863+0000 7fa75c5e2640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa754145eb0 con 0x7fa7540a9d70 2024-03-29T11:30:21.858 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:21.863+0000 7fa75c5e2640 10 monclient(hunting): _renew_subs 2024-03-29T11:30:21.858 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:21.863+0000 7fa75c5e2640 10 monclient(hunting): authenticate will time out at 1483.157681s 2024-03-29T11:30:21.858 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:21.864+0000 7fa75ab7f640 1 --1- >> v1:172.21.15.84:6790/0 conn(0x7fa7540a9d70 0x7fa75414d8c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.84:6790/0 says I am v1:172.21.15.84:56570/0 (socket says 172.21.15.84:56570) 2024-03-29T11:30:21.858 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:21.864+0000 7fa75ab7f640 1 -- 172.21.15.84:0/2047340450 learned_addr learned my addr 172.21.15.84:0/2047340450 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-29T11:30:21.859 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:21.864+0000 7fa75937c640 1 -- 172.21.15.84:0/2047340450 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 916281813 0 0) 0x7fa754145eb0 con 0x7fa7540a9d70 2024-03-29T11:30:21.859 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:21.864+0000 7fa75937c640 10 monclient(hunting): _negotiate 2024-03-29T11:30:21.859 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:21.864+0000 7fa75937c640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:30:21.859 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:21.864+0000 7fa75937c640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:30:21.859 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:21.864+0000 7fa75937c640 10 monclient(hunting): my global_id is 4679 2024-03-29T11:30:21.859 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:21.864+0000 7fa75937c640 1 -- 172.21.15.84:0/2047340450 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa740003520 con 0x7fa7540a9d70 2024-03-29T11:30:21.859 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:21.864+0000 7fa75937c640 1 -- 172.21.15.84:0/2047340450 <== mon.2 v1:172.21.15.84:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2484405781 0 0) 0x7fa740003520 con 0x7fa7540a9d70 2024-03-29T11:30:21.859 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:21.864+0000 7fa75937c640 1 -- 172.21.15.84:0/2047340450 >> v1:172.21.15.179:6789/0 conn(0x7fa7541457f0 legacy=0x7fa75413ef40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:30:21.861 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:21.869+0000 7fa75937c640 1 -- 172.21.15.84:0/2047340450 >> v1:172.21.15.84:6789/0 conn(0x7fa75413f650 legacy=0x7fa7540b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:30:21.861 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:21.869+0000 7fa75937c640 10 monclient: _finish_hunting 0 2024-03-29T11:30:21.861 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:21.869+0000 7fa75937c640 1 monclient: found mon.noname-b 2024-03-29T11:30:21.861 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:21.869+0000 7fa75937c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:30:21.862 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:21.869+0000 7fa75937c640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.84:6790/0 2024-03-29T11:30:21.862 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:21.869+0000 7fa75937c640 1 -- 172.21.15.84:0/2047340450 --> v1:172.21.15.84:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa754146340 con 0x7fa7540a9d70 2024-03-29T11:30:21.862 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:21.869+0000 7fa75937c640 10 monclient: _finish_auth 0 2024-03-29T11:30:21.862 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:21.869+0000 7fa75937c640 10 monclient: _check_auth_tickets 2024-03-29T11:30:21.862 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:21.869+0000 7fa75937c640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:30:21.862 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:21.869+0000 7fa75937c640 1 -- 172.21.15.84:0/2047340450 <== mon.2 v1:172.21.15.84:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4236868694 0 0) 0x7fa74c001a10 con 0x7fa7540a9d70 2024-03-29T11:30:21.862 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:21.869+0000 7fa75937c640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:30:21.862 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:21.869+0000 7fa75937c640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:30:21.862 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:21.869+0000 7fa75937c640 10 monclient: dump: 2024-03-29T11:30:21.862 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T11:30:21.862 INFO:teuthology.misc.health.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:30:21.862 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T11:18:53.864025+0000 2024-03-29T11:30:21.862 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T11:18:53.864025+0000 2024-03-29T11:30:21.862 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T11:30:21.863 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:30:21.863 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T11:30:21.863 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T11:30:21.863 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T11:30:21.863 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:30:21.863 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:21.869+0000 7fa75c5e2640 5 monclient: authenticate success, global_id 4679 2024-03-29T11:30:21.863 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:21.869+0000 7fa75c5e2640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:30:21.864 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:21.871+0000 7fa75937c640 1 -- 172.21.15.84:0/2047340450 <== mon.2 v1:172.21.15.84:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa74c001be0 con 0x7fa7540a9d70 2024-03-29T11:30:21.864 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:21.871+0000 7fa75937c640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:30:21.864 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:21.871+0000 7fa75c5e2640 10 monclient: get_monmap_and_config success 2024-03-29T11:30:21.864 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:21.871+0000 7fa75c5e2640 4 set_mon_vals no callback set 2024-03-29T11:30:21.864 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:21.872+0000 7fa75937c640 1 -- 172.21.15.84:0/2047340450 <== mon.2 v1:172.21.15.84:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4236868694 0 0) 0x7fa74c001ff0 con 0x7fa7540a9d70 2024-03-29T11:30:21.865 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:21.872+0000 7fa75937c640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:30:21.865 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:21.872+0000 7fa75937c640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:30:21.865 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:21.872+0000 7fa75937c640 10 monclient: dump: 2024-03-29T11:30:21.865 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T11:30:21.865 INFO:teuthology.misc.health.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:30:21.865 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T11:18:53.864025+0000 2024-03-29T11:30:21.865 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T11:18:53.864025+0000 2024-03-29T11:30:21.865 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T11:30:21.865 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:30:21.865 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T11:30:21.865 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T11:30:21.866 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T11:30:21.866 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:30:21.866 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:21.872+0000 7fa75c5e2640 10 monclient: shutdown 2024-03-29T11:30:21.866 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:21.872+0000 7fa75c5e2640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:30:21.866 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:21.872+0000 7fa75c5e2640 1 -- 172.21.15.84:0/2047340450 >> v1:172.21.15.84:6790/0 conn(0x7fa7540a9d70 legacy=0x7fa75414d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:30:21.866 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:21.873+0000 7fa75c5e2640 1 -- 172.21.15.84:0/2047340450 shutdown_connections 2024-03-29T11:30:21.866 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:21.873+0000 7fa75c5e2640 1 -- 172.21.15.84:0/2047340450 >> 172.21.15.84:0/2047340450 conn(0x7fa754066a00 msgr2=0x7fa754066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:30:21.867 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:21.875+0000 7fa75c5e2640 1 -- 172.21.15.84:0/2047340450 shutdown_connections 2024-03-29T11:30:21.868 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:21.875+0000 7fa75c5e2640 1 -- 172.21.15.84:0/2047340450 wait complete. 2024-03-29T11:30:21.870 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:21.876+0000 7fa75c5e2640 10 monclient: build_initial_monmap 2024-03-29T11:30:21.870 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:21.876+0000 7fa75c5e2640 10 monclient: monmap: 2024-03-29T11:30:21.870 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-29T11:30:21.870 INFO:teuthology.misc.health.smithi084.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:30:21.870 INFO:teuthology.misc.health.smithi084.stderr:last_changed 0.000000 2024-03-29T11:30:21.871 INFO:teuthology.misc.health.smithi084.stderr:created 0.000000 2024-03-29T11:30:21.871 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-29T11:30:21.871 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:30:21.871 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-29T11:30:21.871 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.noname-b 2024-03-29T11:30:21.871 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-c 2024-03-29T11:30:21.871 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:30:21.871 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:21.876+0000 7fa75c5e2640 1 librados: starting msgr at 2024-03-29T11:30:21.871 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:21.876+0000 7fa75c5e2640 1 librados: starting objecter 2024-03-29T11:30:21.872 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:21.876+0000 7fa75c5e2640 1 Processor -- start 2024-03-29T11:30:21.872 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:21.876+0000 7fa75c5e2640 1 -- start start 2024-03-29T11:30:21.872 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:21.876+0000 7fa75c5e2640 1 librados: setting wanted keys 2024-03-29T11:30:21.872 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:21.876+0000 7fa75c5e2640 1 librados: calling monclient init 2024-03-29T11:30:21.872 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:21.876+0000 7fa75c5e2640 10 monclient: init 2024-03-29T11:30:21.872 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:21.877+0000 7fa75c5e2640 10 monclient: _reopen_session rank -1 2024-03-29T11:30:21.872 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:21.877+0000 7fa75c5e2640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:30:21.872 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:21.877+0000 7fa75c5e2640 10 monclient(hunting): picked mon.noname-a con 0x7fa7540a9d70 addr v1:172.21.15.84:6789/0 2024-03-29T11:30:21.872 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:21.877+0000 7fa75c5e2640 10 monclient(hunting): picked mon.noname-c con 0x7fa75413f650 addr v1:172.21.15.84:6790/0 2024-03-29T11:30:21.872 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:21.877+0000 7fa75c5e2640 10 monclient(hunting): picked mon.noname-b con 0x7fa7541457f0 addr v1:172.21.15.179:6789/0 2024-03-29T11:30:21.872 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:21.877+0000 7fa75c5e2640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa7541e1740 con 0x7fa7540a9d70 2024-03-29T11:30:21.872 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:21.877+0000 7fa75c5e2640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa754142a50 con 0x7fa7541457f0 2024-03-29T11:30:21.872 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:21.877+0000 7fa75c5e2640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa754071ef0 con 0x7fa75413f650 2024-03-29T11:30:21.872 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:21.877+0000 7fa75c5e2640 10 monclient(hunting): _renew_subs 2024-03-29T11:30:21.873 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:21.877+0000 7fa75c5e2640 10 monclient(hunting): authenticate will time out at 1483.171819s 2024-03-29T11:30:21.878 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:21.880+0000 7fa759b7d640 1 --1- >> v1:172.21.15.84:6790/0 conn(0x7fa75413f650 0x7fa7541e3b10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.84:6790/0 says I am v1:172.21.15.84:57004/0 (socket says 172.21.15.84:57004) 2024-03-29T11:30:21.878 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:21.880+0000 7fa759b7d640 1 -- 172.21.15.84:0/1737574330 learned_addr learned my addr 172.21.15.84:0/1737574330 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-29T11:30:21.878 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:21.880+0000 7fa74affd640 10 client.?.objecter ms_handle_connect 0x7fa75413f650 2024-03-29T11:30:21.878 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:21.880+0000 7fa74affd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:30:21.878 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:21.880+0000 7fa74affd640 10 client.?.objecter ms_handle_connect 0x7fa7541457f0 2024-03-29T11:30:21.878 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:21.880+0000 7fa74affd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:30:21.878 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:21.881+0000 7fa74affd640 1 -- 172.21.15.84:0/1737574330 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 647384871 0 0) 0x7fa754071ef0 con 0x7fa75413f650 2024-03-29T11:30:21.879 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:21.881+0000 7fa74affd640 10 monclient(hunting): _negotiate 2024-03-29T11:30:21.879 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:21.881+0000 7fa74affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:30:21.879 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:21.881+0000 7fa74affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:30:21.879 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:21.881+0000 7fa74affd640 10 monclient(hunting): my global_id is 4682 2024-03-29T11:30:21.879 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:21.881+0000 7fa74affd640 1 -- 172.21.15.84:0/1737574330 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa738004eb0 con 0x7fa75413f650 2024-03-29T11:30:21.879 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:21.881+0000 7fa74affd640 1 -- 172.21.15.84:0/1737574330 <== mon.1 v1:172.21.15.179:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1578848709 0 0) 0x7fa754142a50 con 0x7fa7541457f0 2024-03-29T11:30:21.879 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:21.881+0000 7fa74affd640 10 monclient(hunting): _negotiate 2024-03-29T11:30:21.880 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:21.881+0000 7fa74affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:30:21.880 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:21.881+0000 7fa74affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:30:21.880 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:21.881+0000 7fa74affd640 10 monclient(hunting): my global_id is 4549 2024-03-29T11:30:21.880 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:21.881+0000 7fa74affd640 1 -- 172.21.15.84:0/1737574330 --> v1:172.21.15.179:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa754071ef0 con 0x7fa7541457f0 2024-03-29T11:30:21.880 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:21.882+0000 7fa74affd640 1 -- 172.21.15.84:0/1737574330 <== mon.1 v1:172.21.15.179:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2455668698 0 0) 0x7fa754071ef0 con 0x7fa7541457f0 2024-03-29T11:30:21.880 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:21.882+0000 7fa74affd640 1 -- 172.21.15.84:0/1737574330 --> v1:172.21.15.179:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa754142a50 con 0x7fa7541457f0 2024-03-29T11:30:21.880 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:21.882+0000 7fa74affd640 1 -- 172.21.15.84:0/1737574330 <== mon.1 v1:172.21.15.179:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4236868694 0 0) 0x7fa74c001ef0 con 0x7fa7541457f0 2024-03-29T11:30:21.880 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:21.882+0000 7fa74affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:30:21.881 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:21.882+0000 7fa74affd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:30:21.881 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:21.882+0000 7fa74affd640 10 monclient(hunting): dump: 2024-03-29T11:30:21.881 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T11:30:21.881 INFO:teuthology.misc.health.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:30:21.881 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T11:18:53.864025+0000 2024-03-29T11:30:21.881 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T11:18:53.864025+0000 2024-03-29T11:30:21.881 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T11:30:21.882 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:30:21.882 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T11:30:21.882 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T11:30:21.882 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T11:30:21.882 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:30:21.882 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:21.882+0000 7fa74affd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:30:21.883 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:21.882+0000 7fa74affd640 1 -- 172.21.15.84:0/1737574330 <== mon.1 v1:172.21.15.179:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2317225019 0 0) 0x7fa754142a50 con 0x7fa7541457f0 2024-03-29T11:30:21.883 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:21.882+0000 7fa74affd640 1 -- 172.21.15.84:0/1737574330 >> v1:172.21.15.84:6790/0 conn(0x7fa75413f650 legacy=0x7fa7541e3b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:30:21.883 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:21.883+0000 7fa74affd640 1 -- 172.21.15.84:0/1737574330 >> v1:172.21.15.84:6789/0 conn(0x7fa7540a9d70 legacy=0x7fa7540717e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:30:21.883 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:21.884+0000 7fa74affd640 10 monclient: _finish_hunting 0 2024-03-29T11:30:21.883 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:21.884+0000 7fa74affd640 1 monclient: found mon.b 2024-03-29T11:30:21.883 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:21.884+0000 7fa74affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:30:21.883 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:21.884+0000 7fa74affd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-29T11:30:21.883 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:21.884+0000 7fa74affd640 1 -- 172.21.15.84:0/1737574330 --> v1:172.21.15.179:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa754072080 con 0x7fa7541457f0 2024-03-29T11:30:21.883 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:21.884+0000 7fa74affd640 10 monclient: _finish_auth 0 2024-03-29T11:30:21.883 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:21.884+0000 7fa74affd640 10 monclient: _check_auth_tickets 2024-03-29T11:30:21.883 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:21.884+0000 7fa74affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:30:21.884 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:21.884+0000 7fa75c5e2640 5 monclient: authenticate success, global_id 4549 2024-03-29T11:30:21.884 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:21.884+0000 7fa75c5e2640 10 monclient: _renew_subs 2024-03-29T11:30:21.884 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:21.884+0000 7fa75c5e2640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-29T11:30:21.884 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:21.884+0000 7fa75c5e2640 1 -- 172.21.15.84:0/1737574330 --> v1:172.21.15.179:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa75406e9a0 con 0x7fa7541457f0 2024-03-29T11:30:21.884 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:21.884+0000 7fa75c5e2640 10 client.4549.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:30:21.884 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:21.884+0000 7fa75c5e2640 10 monclient: _renew_subs 2024-03-29T11:30:21.884 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:21.884+0000 7fa75c5e2640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-29T11:30:21.884 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:21.884+0000 7fa75c5e2640 1 -- 172.21.15.84:0/1737574330 --> v1:172.21.15.179:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa754139710 con 0x7fa7541457f0 2024-03-29T11:30:21.884 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:21.884+0000 7fa75c5e2640 1 librados: init done 2024-03-29T11:30:21.887 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:21.886+0000 7fa74affd640 1 -- 172.21.15.84:0/1737574330 <== mon.1 v1:172.21.15.179:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa74c001390 con 0x7fa7541457f0 2024-03-29T11:30:21.887 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:21.886+0000 7fa74affd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:30:21.887 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:21.886+0000 7fa75c5e2640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:30:21.887 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:21.887+0000 7fa74affd640 1 -- 172.21.15.84:0/1737574330 <== mon.1 v1:172.21.15.179:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4236868694 0 0) 0x7fa74c004f70 con 0x7fa7541457f0 2024-03-29T11:30:21.888 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:21.887+0000 7fa74affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:30:21.888 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:21.887+0000 7fa74affd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:30:21.888 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:21.887+0000 7fa74affd640 10 monclient: dump: 2024-03-29T11:30:21.888 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T11:30:21.888 INFO:teuthology.misc.health.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:30:21.888 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T11:18:53.864025+0000 2024-03-29T11:30:21.888 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T11:18:53.864025+0000 2024-03-29T11:30:21.888 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T11:30:21.888 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:30:21.888 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T11:30:21.888 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T11:30:21.888 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T11:30:21.888 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:30:21.889 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:21.887+0000 7fa75c5e2640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:30:21.889 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:21.887+0000 7fa75c5e2640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-29T11:30:21.889 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:21.887+0000 7fa75c5e2640 1 -- 172.21.15.84:0/1737574330 --> v1:172.21.15.179:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa7541428b0 con 0x7fa7541457f0 2024-03-29T11:30:21.889 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:21.887+0000 7fa74bfff640 4 set_mon_vals no callback set 2024-03-29T11:30:21.889 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:21.887+0000 7fa74affd640 1 -- 172.21.15.84:0/1737574330 <== mon.1 v1:172.21.15.179:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4033942909 0 0) 0x7fa74c01af40 con 0x7fa7541457f0 2024-03-29T11:30:21.889 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:21.888+0000 7fa74affd640 1 -- 172.21.15.84:0/1737574330 <== mon.1 v1:172.21.15.179:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 156031568 0 0) 0x7fa74c084d70 con 0x7fa7541457f0 2024-03-29T11:30:21.889 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:21.888+0000 7fa74affd640 10 client.4549.objecter ms_dispatch 0x7fa754000b90 osd_map(11..11 src has 1..11) v4 2024-03-29T11:30:21.889 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:21.888+0000 7fa74affd640 3 client.4549.objecter handle_osd_map got epochs [11,11] > 0 2024-03-29T11:30:21.889 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:21.888+0000 7fa74affd640 3 client.4549.objecter handle_osd_map decoding full epoch 11 2024-03-29T11:30:21.889 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:21.888+0000 7fa74affd640 20 client.4549.objecter dump_active .. 0 homeless 2024-03-29T11:30:21.889 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:21.889+0000 7fa75a37e640 10 monclient: get_auth_request con 0x7fa73806dd70 auth_method 0 2024-03-29T11:30:21.890 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:21.889+0000 7fa74affd640 10 client.4549.objecter ms_handle_connect 0x7fa73806dd70 2024-03-29T11:30:21.890 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:21.895+0000 7fa74affd640 1 -- 172.21.15.84:0/1737574330 <== mon.1 v1:172.21.15.179:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fa7541428b0 con 0x7fa7541457f0 2024-03-29T11:30:21.891 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:21.896+0000 7fa74affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:30:21.891 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:21.896+0000 7fa74affd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:30:22.250 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:22.257+0000 7fa75c5e2640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:30:22.250 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:22.257+0000 7fa75c5e2640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:30:22.250 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:22.257+0000 7fa75c5e2640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-29T11:30:22.250 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:22.257+0000 7fa75c5e2640 1 -- 172.21.15.84:0/1737574330 --> v1:172.21.15.179:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa7541428b0 con 0x7fa7541457f0 2024-03-29T11:30:22.251 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:22.258+0000 7fa74affd640 1 -- 172.21.15.84:0/1737574330 <== mon.1 v1:172.21.15.179:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 782149409) 0x7fa7541428b0 con 0x7fa7541457f0 2024-03-29T11:30:22.251 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:22.258+0000 7fa74affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:30:22.251 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:22.258+0000 7fa74affd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:30:22.251 INFO:teuthology.misc.health.smithi084.stdout: 2024-03-29T11:30:22.251 INFO:teuthology.misc.health.smithi084.stdout:{"epoch":11,"fsid":"2c3e002f-9eb0-4860-853b-4ef1ef562213","created":"2024-03-29T11:19:32.029169+0000","modified":"2024-03-29T11:29:36.955202+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:29:36.955202+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"5c39e276-2034-4716-bbc0-57a634eefa92","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"cd580e99-109f-4bf1-88c0-a5987bc8b9ed","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"11e60a5a-051d-453f-a190-66387cb35592","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"2c329fbe-bbc4-4030-bf87-d711cad04679","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f4213d4a-bd36-4433-91ae-d7e2b02481d3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"f0cf44ea-6055-48af-805b-aed858102ff9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"d895eaa0-cc0b-4ee4-9b5a-2dfcd89a60b7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"9bc5a7bc-8f52-4db9-96d6-de5e68d52eaa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:30:22.254 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:22.262+0000 7fa75c5e2640 10 librados: watch_flush enter 2024-03-29T11:30:22.255 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:22.262+0000 7fa75c5e2640 10 librados: watch_flush exit 2024-03-29T11:30:22.255 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:22.262+0000 7fa75c5e2640 20 client.4549.objecter shutdown clearing up homeless session... 2024-03-29T11:30:22.255 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:22.262+0000 7fa75c5e2640 10 client.4549.objecter successfully canceled tick 2024-03-29T11:30:22.255 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:22.262+0000 7fa75c5e2640 1 -- 172.21.15.84:0/1737574330 >> v1:172.21.15.179:6816/2981134980 conn(0x7fa73806dd70 legacy=0x7fa738070230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:30:22.255 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:22.262+0000 7fa75c5e2640 10 monclient: shutdown 2024-03-29T11:30:22.255 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:22.262+0000 7fa75c5e2640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:30:22.255 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:22.262+0000 7fa75c5e2640 1 -- 172.21.15.84:0/1737574330 >> v1:172.21.15.179:6789/0 conn(0x7fa7541457f0 legacy=0x7fa7541ef3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:30:22.255 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:22.262+0000 7fa75c5e2640 1 -- 172.21.15.84:0/1737574330 shutdown_connections 2024-03-29T11:30:22.255 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:22.262+0000 7fa75c5e2640 1 -- 172.21.15.84:0/1737574330 >> 172.21.15.84:0/1737574330 conn(0x7fa754066a00 msgr2=0x7fa7540b4890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:30:22.255 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:22.263+0000 7fa75c5e2640 1 -- 172.21.15.84:0/1737574330 shutdown_connections 2024-03-29T11:30:22.258 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:22.265+0000 7fa75c5e2640 1 -- 172.21.15.84:0/1737574330 wait complete. 2024-03-29T11:30:22.259 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:22.266+0000 7fa75c5e2640 1 librados: shutdown 2024-03-29T11:30:22.278 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:30:29.085 DEBUG:teuthology.orchestra.run.smithi084:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:30:29.237 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:29.244+0000 7f900d59d640 10 monclient: get_monmap_and_config 2024-03-29T11:30:29.238 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:29.244+0000 7f900d59d640 10 monclient: build_initial_monmap 2024-03-29T11:30:29.238 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:29.245+0000 7f900d59d640 10 monclient: monmap: 2024-03-29T11:30:29.238 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-29T11:30:29.238 INFO:teuthology.misc.health.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:30:29.238 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T11:30:29.245914+0000 2024-03-29T11:30:29.238 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T11:30:29.245914+0000 2024-03-29T11:30:29.238 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-29T11:30:29.238 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:30:29.238 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-29T11:30:29.238 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.noname-c 2024-03-29T11:30:29.238 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-b 2024-03-29T11:30:29.238 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:30:29.240 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:29.246+0000 7f900d59d640 1 Processor -- start 2024-03-29T11:30:29.240 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:29.246+0000 7f900d59d640 1 -- start start 2024-03-29T11:30:29.240 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:29.246+0000 7f900d59d640 10 monclient: init 2024-03-29T11:30:29.240 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:29.247+0000 7f900d59d640 10 monclient: _reopen_session rank -1 2024-03-29T11:30:29.240 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:29.247+0000 7f900d59d640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:30:29.240 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:29.247+0000 7f900d59d640 10 monclient(hunting): picked mon.noname-c con 0x7f900806ade0 addr v1:172.21.15.179:6789/0 2024-03-29T11:30:29.240 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:29.247+0000 7f900d59d640 10 monclient(hunting): picked mon.noname-a con 0x7f900806ed60 addr v1:172.21.15.84:6789/0 2024-03-29T11:30:29.240 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:29.247+0000 7f900d59d640 10 monclient(hunting): picked mon.noname-b con 0x7f9008072340 addr v1:172.21.15.84:6790/0 2024-03-29T11:30:29.242 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:29.247+0000 7f900d59d640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f900806ff50 con 0x7f900806ed60 2024-03-29T11:30:29.243 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:29.247+0000 7f900d59d640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9008070180 con 0x7f900806ade0 2024-03-29T11:30:29.243 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:29.247+0000 7f900d59d640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f900806f310 con 0x7f9008072340 2024-03-29T11:30:29.243 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:29.247+0000 7f900d59d640 10 monclient(hunting): _renew_subs 2024-03-29T11:30:29.243 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:29.247+0000 7f900d59d640 10 monclient(hunting): authenticate will time out at 1490.541225s 2024-03-29T11:30:29.243 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:29.247+0000 7f90067fc640 1 --1- >> v1:172.21.15.84:6789/0 conn(0x7f900806ed60 0x7f9008071c30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.84:6789/0 says I am v1:172.21.15.84:52982/0 (socket says 172.21.15.84:52982) 2024-03-29T11:30:29.243 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:29.247+0000 7f90067fc640 1 -- 172.21.15.84:0/4101226952 learned_addr learned my addr 172.21.15.84:0/4101226952 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-29T11:30:29.243 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:29.248+0000 7f9005ffb640 1 -- 172.21.15.84:0/4101226952 <== mon.0 v1:172.21.15.84:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 171362695 0 0) 0x7f900806ff50 con 0x7f900806ed60 2024-03-29T11:30:29.243 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:29.248+0000 7f9005ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:30:29.243 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:29.248+0000 7f9005ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:30:29.243 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:29.248+0000 7f9005ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:30:29.243 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:29.248+0000 7f9005ffb640 10 monclient(hunting): my global_id is 4611 2024-03-29T11:30:29.243 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:29.248+0000 7f9005ffb640 1 -- 172.21.15.84:0/4101226952 --> v1:172.21.15.84:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8ff0003520 con 0x7f900806ed60 2024-03-29T11:30:29.243 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:29.248+0000 7f9005ffb640 1 -- 172.21.15.84:0/4101226952 <== mon.1 v1:172.21.15.179:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1992683021 0 0) 0x7f9008070180 con 0x7f900806ade0 2024-03-29T11:30:29.243 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:29.248+0000 7f9005ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:30:29.244 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:29.248+0000 7f9005ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:30:29.244 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:29.248+0000 7f9005ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:30:29.244 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:29.248+0000 7f9005ffb640 10 monclient(hunting): my global_id is 4552 2024-03-29T11:30:29.244 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:29.248+0000 7f9005ffb640 1 -- 172.21.15.84:0/4101226952 --> v1:172.21.15.179:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f900806ff50 con 0x7f900806ade0 2024-03-29T11:30:29.244 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:29.248+0000 7f9005ffb640 1 -- 172.21.15.84:0/4101226952 <== mon.0 v1:172.21.15.84:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2354876545 0 0) 0x7f8ff0003520 con 0x7f900806ed60 2024-03-29T11:30:29.244 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:29.249+0000 7f9005ffb640 1 -- 172.21.15.84:0/4101226952 >> v1:172.21.15.84:6790/0 conn(0x7f9008072340 legacy=0x7f9008151d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:30:29.244 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:29.249+0000 7f9005ffb640 1 -- 172.21.15.84:0/4101226952 >> v1:172.21.15.179:6789/0 conn(0x7f900806ade0 legacy=0x7f90081499e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:30:29.244 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:29.249+0000 7f9005ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:30:29.244 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:29.249+0000 7f9005ffb640 1 monclient: found mon.noname-a 2024-03-29T11:30:29.244 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:29.249+0000 7f9005ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:30:29.244 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:29.249+0000 7f9005ffb640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.84:6789/0 2024-03-29T11:30:29.244 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:29.249+0000 7f9005ffb640 1 -- 172.21.15.84:0/4101226952 --> v1:172.21.15.84:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90080aeed0 con 0x7f900806ed60 2024-03-29T11:30:29.244 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:29.249+0000 7f9005ffb640 10 monclient: _finish_auth 0 2024-03-29T11:30:29.244 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:29.249+0000 7f9005ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:30:29.244 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:29.249+0000 7f9005ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:30:29.245 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:29.249+0000 7f9005ffb640 1 -- 172.21.15.84:0/4101226952 <== mon.0 v1:172.21.15.84:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4236868694 0 0) 0x7f8ff8001a10 con 0x7f900806ed60 2024-03-29T11:30:29.245 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:29.249+0000 7f9005ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:30:29.245 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:29.249+0000 7f9005ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:30:29.245 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:29.249+0000 7f9005ffb640 10 monclient: dump: 2024-03-29T11:30:29.245 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T11:30:29.245 INFO:teuthology.misc.health.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:30:29.245 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T11:18:53.864025+0000 2024-03-29T11:30:29.245 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T11:18:53.864025+0000 2024-03-29T11:30:29.245 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T11:30:29.245 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:30:29.245 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T11:30:29.245 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T11:30:29.245 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T11:30:29.245 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:30:29.245 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:29.249+0000 7f900d59d640 5 monclient: authenticate success, global_id 4611 2024-03-29T11:30:29.246 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:29.249+0000 7f900d59d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:30:29.246 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:29.249+0000 7f9005ffb640 1 -- 172.21.15.84:0/4101226952 <== mon.0 v1:172.21.15.84:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8ff8001b70 con 0x7f900806ed60 2024-03-29T11:30:29.246 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:29.249+0000 7f9005ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:30:29.246 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:29.249+0000 7f900d59d640 10 monclient: get_monmap_and_config success 2024-03-29T11:30:29.246 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:29.249+0000 7f900d59d640 4 set_mon_vals no callback set 2024-03-29T11:30:29.246 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:29.249+0000 7f9005ffb640 1 -- 172.21.15.84:0/4101226952 <== mon.0 v1:172.21.15.84:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4236868694 0 0) 0x7f8ff8001f80 con 0x7f900806ed60 2024-03-29T11:30:29.246 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:29.249+0000 7f9005ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:30:29.246 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:29.249+0000 7f9005ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:30:29.246 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:29.249+0000 7f9005ffb640 10 monclient: dump: 2024-03-29T11:30:29.246 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T11:30:29.246 INFO:teuthology.misc.health.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:30:29.246 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T11:18:53.864025+0000 2024-03-29T11:30:29.246 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T11:18:53.864025+0000 2024-03-29T11:30:29.246 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T11:30:29.246 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:30:29.247 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T11:30:29.247 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T11:30:29.247 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T11:30:29.247 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:30:29.247 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:29.250+0000 7f900d59d640 10 monclient: shutdown 2024-03-29T11:30:29.247 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:29.250+0000 7f900d59d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:30:29.247 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:29.250+0000 7f900d59d640 1 -- 172.21.15.84:0/4101226952 >> v1:172.21.15.84:6789/0 conn(0x7f900806ed60 legacy=0x7f9008071c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:30:29.251 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:29.250+0000 7f900d59d640 1 -- 172.21.15.84:0/4101226952 shutdown_connections 2024-03-29T11:30:29.251 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:29.250+0000 7f900d59d640 1 -- 172.21.15.84:0/4101226952 >> 172.21.15.84:0/4101226952 conn(0x7f90080b4930 msgr2=0x7f90080b4d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:30:29.252 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:29.250+0000 7f900d59d640 1 -- 172.21.15.84:0/4101226952 shutdown_connections 2024-03-29T11:30:29.252 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:29.250+0000 7f900d59d640 1 -- 172.21.15.84:0/4101226952 wait complete. 2024-03-29T11:30:29.252 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:29.251+0000 7f900d59d640 10 monclient: build_initial_monmap 2024-03-29T11:30:29.252 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:29.251+0000 7f900d59d640 10 monclient: monmap: 2024-03-29T11:30:29.252 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-29T11:30:29.252 INFO:teuthology.misc.health.smithi084.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:30:29.252 INFO:teuthology.misc.health.smithi084.stderr:last_changed 0.000000 2024-03-29T11:30:29.252 INFO:teuthology.misc.health.smithi084.stderr:created 0.000000 2024-03-29T11:30:29.252 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-29T11:30:29.252 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:30:29.252 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-29T11:30:29.252 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.noname-b 2024-03-29T11:30:29.252 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-c 2024-03-29T11:30:29.253 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:30:29.253 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:29.251+0000 7f900d59d640 1 librados: starting msgr at 2024-03-29T11:30:29.253 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:29.251+0000 7f900d59d640 1 librados: starting objecter 2024-03-29T11:30:29.253 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:29.252+0000 7f900d59d640 1 Processor -- start 2024-03-29T11:30:29.253 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:29.252+0000 7f900d59d640 1 -- start start 2024-03-29T11:30:29.253 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:29.252+0000 7f900d59d640 1 librados: setting wanted keys 2024-03-29T11:30:29.253 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:29.252+0000 7f900d59d640 1 librados: calling monclient init 2024-03-29T11:30:29.253 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:29.252+0000 7f900d59d640 10 monclient: init 2024-03-29T11:30:29.253 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:29.253+0000 7f900d59d640 10 monclient: _reopen_session rank -1 2024-03-29T11:30:29.253 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:29.253+0000 7f900d59d640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:30:29.253 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:29.253+0000 7f900d59d640 10 monclient(hunting): picked mon.noname-b con 0x7f900806ade0 addr v1:172.21.15.179:6789/0 2024-03-29T11:30:29.253 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:29.253+0000 7f900d59d640 10 monclient(hunting): picked mon.noname-c con 0x7f900806ed60 addr v1:172.21.15.84:6790/0 2024-03-29T11:30:29.253 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:29.253+0000 7f900d59d640 10 monclient(hunting): picked mon.noname-a con 0x7f9008072340 addr v1:172.21.15.84:6789/0 2024-03-29T11:30:29.253 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:29.253+0000 7f90077fe640 1 --1- >> v1:172.21.15.84:6789/0 conn(0x7f9008072340 0x7f90080adbc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.84:6789/0 says I am v1:172.21.15.84:56612/0 (socket says 172.21.15.84:56612) 2024-03-29T11:30:29.253 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:29.253+0000 7f90077fe640 1 -- 172.21.15.84:0/1769983653 learned_addr learned my addr 172.21.15.84:0/1769983653 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-29T11:30:29.254 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:29.253+0000 7f900d59d640 1 -- 172.21.15.84:0/1769983653 --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f90081de9f0 con 0x7f9008072340 2024-03-29T11:30:29.254 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:29.253+0000 7f900d59d640 1 -- 172.21.15.84:0/1769983653 --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f90080ac230 con 0x7f900806ade0 2024-03-29T11:30:29.254 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:29.254+0000 7f900d59d640 1 -- 172.21.15.84:0/1769983653 --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f90081454c0 con 0x7f900806ed60 2024-03-29T11:30:29.254 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:29.254+0000 7f900d59d640 10 monclient(hunting): _renew_subs 2024-03-29T11:30:29.254 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:29.254+0000 7f900d59d640 10 monclient(hunting): authenticate will time out at 1490.548188s 2024-03-29T11:30:29.254 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:29.254+0000 7f8fef7fe640 10 client.?.objecter ms_handle_connect 0x7f900806ed60 2024-03-29T11:30:29.254 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:29.254+0000 7f8fef7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:30:29.254 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:29.254+0000 7f8fef7fe640 10 client.?.objecter ms_handle_connect 0x7f9008072340 2024-03-29T11:30:29.254 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:29.254+0000 7f8fef7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:30:29.254 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:29.254+0000 7f8fef7fe640 1 -- 172.21.15.84:0/1769983653 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1432674777 0 0) 0x7f90081454c0 con 0x7f900806ed60 2024-03-29T11:30:29.254 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:29.254+0000 7f8fef7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:30:29.254 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:29.254+0000 7f8fef7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:30:29.254 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:29.254+0000 7f8fef7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:30:29.254 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:29.254+0000 7f8fef7fe640 10 monclient(hunting): my global_id is 4688 2024-03-29T11:30:29.255 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:29.254+0000 7f8fef7fe640 1 -- 172.21.15.84:0/1769983653 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8ff4004eb0 con 0x7f900806ed60 2024-03-29T11:30:29.255 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:29.254+0000 7f8fef7fe640 1 -- 172.21.15.84:0/1769983653 <== mon.0 v1:172.21.15.84:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 808399343 0 0) 0x7f90081de9f0 con 0x7f9008072340 2024-03-29T11:30:29.255 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:29.254+0000 7f8fef7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:30:29.255 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:29.254+0000 7f8fef7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:30:29.255 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:29.254+0000 7f8fef7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:30:29.255 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:29.254+0000 7f8fef7fe640 10 monclient(hunting): my global_id is 4614 2024-03-29T11:30:29.255 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:29.254+0000 7f8fef7fe640 1 -- 172.21.15.84:0/1769983653 --> v1:172.21.15.84:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f90081454c0 con 0x7f9008072340 2024-03-29T11:30:29.255 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:29.255+0000 7f8fef7fe640 10 client.?.objecter ms_handle_connect 0x7f900806ade0 2024-03-29T11:30:29.255 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:29.255+0000 7f8fef7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:30:29.255 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:29.255+0000 7f8fef7fe640 1 -- 172.21.15.84:0/1769983653 <== mon.2 v1:172.21.15.84:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1929092289 0 0) 0x7f8ff4004eb0 con 0x7f900806ed60 2024-03-29T11:30:29.255 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:29.255+0000 7f8fef7fe640 1 -- 172.21.15.84:0/1769983653 --> v1:172.21.15.84:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f90081de9f0 con 0x7f900806ed60 2024-03-29T11:30:29.255 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:29.255+0000 7f8fef7fe640 1 -- 172.21.15.84:0/1769983653 <== mon.2 v1:172.21.15.84:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4236868694 0 0) 0x7f8ff8001e80 con 0x7f900806ed60 2024-03-29T11:30:29.255 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:29.255+0000 7f8fef7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:30:29.255 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:29.255+0000 7f8fef7fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:30:29.255 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:29.255+0000 7f8fef7fe640 10 monclient(hunting): dump: 2024-03-29T11:30:29.255 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T11:30:29.256 INFO:teuthology.misc.health.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:30:29.256 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T11:18:53.864025+0000 2024-03-29T11:30:29.256 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T11:18:53.864025+0000 2024-03-29T11:30:29.256 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T11:30:29.256 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:30:29.256 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T11:30:29.256 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T11:30:29.256 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T11:30:29.256 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:30:29.256 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:29.255+0000 7f8fef7fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:30:29.256 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:29.255+0000 7f8fef7fe640 1 -- 172.21.15.84:0/1769983653 <== mon.0 v1:172.21.15.84:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 113902773 0 0) 0x7f90081454c0 con 0x7f9008072340 2024-03-29T11:30:29.256 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:29.255+0000 7f8fef7fe640 1 -- 172.21.15.84:0/1769983653 --> v1:172.21.15.84:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8ff4004eb0 con 0x7f9008072340 2024-03-29T11:30:29.256 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:29.255+0000 7f8fef7fe640 1 -- 172.21.15.84:0/1769983653 <== mon.0 v1:172.21.15.84:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4236868694 0 0) 0x7f90000017f0 con 0x7f9008072340 2024-03-29T11:30:29.256 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:29.255+0000 7f8fef7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:30:29.256 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:29.255+0000 7f8fef7fe640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:30:29.257 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:29.255+0000 7f8fef7fe640 10 monclient(hunting): dump: 2024-03-29T11:30:29.257 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T11:30:29.257 INFO:teuthology.misc.health.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:30:29.257 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T11:18:53.864025+0000 2024-03-29T11:30:29.257 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T11:18:53.864025+0000 2024-03-29T11:30:29.257 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T11:30:29.257 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:30:29.257 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T11:30:29.257 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T11:30:29.257 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T11:30:29.257 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:30:29.257 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:29.256+0000 7f8fef7fe640 1 -- 172.21.15.84:0/1769983653 <== mon.2 v1:172.21.15.84:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3158664900 0 0) 0x7f90081de9f0 con 0x7f900806ed60 2024-03-29T11:30:29.257 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:29.256+0000 7f8fef7fe640 1 -- 172.21.15.84:0/1769983653 >> v1:172.21.15.179:6789/0 conn(0x7f900806ade0 legacy=0x7f90080aaa40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:30:29.257 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:29.256+0000 7f8fef7fe640 1 -- 172.21.15.84:0/1769983653 >> v1:172.21.15.84:6789/0 conn(0x7f9008072340 legacy=0x7f90080adbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:30:29.257 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:29.256+0000 7f8fef7fe640 10 monclient: _finish_hunting 0 2024-03-29T11:30:29.257 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:29.256+0000 7f8fef7fe640 1 monclient: found mon.c 2024-03-29T11:30:29.258 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:29.256+0000 7f8fef7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:30:29.258 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:29.256+0000 7f8fef7fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-29T11:30:29.258 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:29.256+0000 7f8fef7fe640 1 -- 172.21.15.84:0/1769983653 --> v1:172.21.15.84:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90080ae2d0 con 0x7f900806ed60 2024-03-29T11:30:29.258 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:29.256+0000 7f8fef7fe640 10 monclient: _finish_auth 0 2024-03-29T11:30:29.258 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:29.256+0000 7f8fef7fe640 10 monclient: _check_auth_tickets 2024-03-29T11:30:29.258 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:29.256+0000 7f8fef7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:30:29.258 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:29.256+0000 7f900d59d640 5 monclient: authenticate success, global_id 4688 2024-03-29T11:30:29.258 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:29.256+0000 7f900d59d640 10 monclient: _renew_subs 2024-03-29T11:30:29.258 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:29.256+0000 7f900d59d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-29T11:30:29.258 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:29.256+0000 7f900d59d640 1 -- 172.21.15.84:0/1769983653 --> v1:172.21.15.84:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f90080ae580 con 0x7f900806ed60 2024-03-29T11:30:29.258 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:29.256+0000 7f900d59d640 10 client.4688.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:30:29.258 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:29.256+0000 7f900d59d640 10 monclient: _renew_subs 2024-03-29T11:30:29.258 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:29.256+0000 7f900d59d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-29T11:30:29.258 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:29.256+0000 7f900d59d640 1 -- 172.21.15.84:0/1769983653 --> v1:172.21.15.84:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f90081df0b0 con 0x7f900806ed60 2024-03-29T11:30:29.259 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:29.256+0000 7f900d59d640 1 librados: init done 2024-03-29T11:30:29.259 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:29.257+0000 7f8fef7fe640 1 -- 172.21.15.84:0/1769983653 <== mon.2 v1:172.21.15.84:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8ff80015a0 con 0x7f900806ed60 2024-03-29T11:30:29.259 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:29.257+0000 7f8fef7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:30:29.259 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:29.257+0000 7f90057fa640 4 set_mon_vals no callback set 2024-03-29T11:30:29.259 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:29.257+0000 7f900d59d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:30:29.259 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:29.258+0000 7f8fef7fe640 1 -- 172.21.15.84:0/1769983653 <== mon.2 v1:172.21.15.84:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4236868694 0 0) 0x7f8ff8004fb0 con 0x7f900806ed60 2024-03-29T11:30:29.259 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:29.258+0000 7f8fef7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:30:29.259 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:29.258+0000 7f8fef7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:30:29.259 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:29.258+0000 7f8fef7fe640 10 monclient: dump: 2024-03-29T11:30:29.259 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T11:30:29.259 INFO:teuthology.misc.health.smithi084.stderr:fsid 2c3e002f-9eb0-4860-853b-4ef1ef562213 2024-03-29T11:30:29.259 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T11:18:53.864025+0000 2024-03-29T11:30:29.259 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T11:18:53.864025+0000 2024-03-29T11:30:29.259 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T11:30:29.259 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T11:30:29.260 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T11:30:29.260 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T11:30:29.260 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T11:30:29.260 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T11:30:29.260 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:29.259+0000 7f900d59d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:30:29.260 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:29.259+0000 7f900d59d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-29T11:30:29.260 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:29.259+0000 7f900d59d640 1 -- 172.21.15.84:0/1769983653 --> v1:172.21.15.84:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8fe8005080 con 0x7f900806ed60 2024-03-29T11:30:29.260 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:29.259+0000 7f8fef7fe640 1 -- 172.21.15.84:0/1769983653 <== mon.2 v1:172.21.15.84:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4033942909 0 0) 0x7f8ff8005110 con 0x7f900806ed60 2024-03-29T11:30:29.261 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:29.260+0000 7f9006ffd640 10 monclient: get_auth_request con 0x7f8ff406de80 auth_method 0 2024-03-29T11:30:29.261 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:29.260+0000 7f8fef7fe640 1 -- 172.21.15.84:0/1769983653 <== mon.2 v1:172.21.15.84:6790/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 156031568 0 0) 0x7f8ff8084e20 con 0x7f900806ed60 2024-03-29T11:30:29.261 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:29.260+0000 7f8fef7fe640 10 client.4688.objecter ms_dispatch 0x7f9008000b90 osd_map(11..11 src has 1..11) v4 2024-03-29T11:30:29.261 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:29.260+0000 7f8fef7fe640 3 client.4688.objecter handle_osd_map got epochs [11,11] > 0 2024-03-29T11:30:29.261 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:29.260+0000 7f8fef7fe640 3 client.4688.objecter handle_osd_map decoding full epoch 11 2024-03-29T11:30:29.261 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:29.260+0000 7f8fef7fe640 20 client.4688.objecter dump_active .. 0 homeless 2024-03-29T11:30:29.261 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:29.260+0000 7f8fef7fe640 10 client.4688.objecter ms_handle_connect 0x7f8ff406de80 2024-03-29T11:30:29.261 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:29.267+0000 7f8fef7fe640 1 -- 172.21.15.84:0/1769983653 <== mon.2 v1:172.21.15.84:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f8ff8050530 con 0x7f900806ed60 2024-03-29T11:30:29.261 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:29.267+0000 7f8fef7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:30:29.261 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:29.267+0000 7f8fef7fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:30:29.584 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:29.591+0000 7f900d59d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:30:29.584 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:29.591+0000 7f900d59d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:30:29.584 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:29.591+0000 7f900d59d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-29T11:30:29.584 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:29.591+0000 7f900d59d640 1 -- 172.21.15.84:0/1769983653 --> v1:172.21.15.84:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8fe8005330 con 0x7f900806ed60 2024-03-29T11:30:29.588 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:29.595+0000 7f8fef7fe640 1 -- 172.21.15.84:0/1769983653 <== mon.2 v1:172.21.15.84:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 782149409) 0x7f8ff8052630 con 0x7f900806ed60 2024-03-29T11:30:29.588 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:29.595+0000 7f8fef7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:30:29.588 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:29.595+0000 7f8fef7fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:30:29.588 INFO:teuthology.misc.health.smithi084.stdout: 2024-03-29T11:30:29.588 INFO:teuthology.misc.health.smithi084.stdout:{"epoch":11,"fsid":"2c3e002f-9eb0-4860-853b-4ef1ef562213","created":"2024-03-29T11:19:32.029169+0000","modified":"2024-03-29T11:29:36.955202+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:29:36.955202+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"5c39e276-2034-4716-bbc0-57a634eefa92","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"cd580e99-109f-4bf1-88c0-a5987bc8b9ed","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"11e60a5a-051d-453f-a190-66387cb35592","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"2c329fbe-bbc4-4030-bf87-d711cad04679","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f4213d4a-bd36-4433-91ae-d7e2b02481d3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"f0cf44ea-6055-48af-805b-aed858102ff9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"d895eaa0-cc0b-4ee4-9b5a-2dfcd89a60b7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"9bc5a7bc-8f52-4db9-96d6-de5e68d52eaa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:30:29.591 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:29.598+0000 7f900d59d640 10 librados: watch_flush enter 2024-03-29T11:30:29.591 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:29.598+0000 7f900d59d640 10 librados: watch_flush exit 2024-03-29T11:30:29.591 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:29.598+0000 7f900d59d640 20 client.4688.objecter shutdown clearing up homeless session... 2024-03-29T11:30:29.591 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:29.598+0000 7f900d59d640 10 client.4688.objecter successfully canceled tick 2024-03-29T11:30:29.592 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:29.599+0000 7f900d59d640 1 -- 172.21.15.84:0/1769983653 >> v1:172.21.15.179:6816/2981134980 conn(0x7f8ff406de80 legacy=0x7f8ff4070340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:30:29.592 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:29.599+0000 7f900d59d640 10 monclient: shutdown 2024-03-29T11:30:29.592 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:29.599+0000 7f900d59d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:30:29.592 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:29.599+0000 7f900d59d640 1 -- 172.21.15.84:0/1769983653 >> v1:172.21.15.84:6790/0 conn(0x7f900806ed60 legacy=0x7f90080a9dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:30:29.592 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:29.599+0000 7f900d59d640 1 -- 172.21.15.84:0/1769983653 shutdown_connections 2024-03-29T11:30:29.592 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:29.599+0000 7f900d59d640 1 -- 172.21.15.84:0/1769983653 >> 172.21.15.84:0/1769983653 conn(0x7f90080b4930 msgr2=0x7f90080afa80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:30:29.592 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:29.600+0000 7f900d59d640 1 -- 172.21.15.84:0/1769983653 shutdown_connections 2024-03-29T11:30:29.593 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:29.600+0000 7f900d59d640 1 -- 172.21.15.84:0/1769983653 wait complete. 2024-03-29T11:30:29.593 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T11:30:29.600+0000 7f900d59d640 1 librados: shutdown 2024-03-29T11:30:29.611 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:30:29.611 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds 2024-03-29T11:30:29.613 INFO:teuthology.misc:Shutting down osd daemons... 2024-03-29T11:30:29.613 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2024-03-29T11:30:29.613 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:30:29.648 INFO:tasks.ceph.osd.0:Stopped 2024-03-29T11:30:29.648 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2024-03-29T11:30:29.648 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:30:29.723 INFO:tasks.ceph.osd.1:Stopped 2024-03-29T11:30:29.723 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2024-03-29T11:30:29.723 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:30:29.791 INFO:tasks.ceph.osd.2:Stopped 2024-03-29T11:30:29.791 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2024-03-29T11:30:29.791 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:30:29.867 INFO:tasks.ceph.osd.3:Stopped 2024-03-29T11:30:29.867 DEBUG:tasks.ceph.osd.4:waiting for process to exit 2024-03-29T11:30:29.867 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:30:29.900 INFO:tasks.ceph.osd.4:Stopped 2024-03-29T11:30:29.900 DEBUG:tasks.ceph.osd.5:waiting for process to exit 2024-03-29T11:30:29.901 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:30:29.971 INFO:tasks.ceph.osd.5:Stopped 2024-03-29T11:30:29.971 DEBUG:tasks.ceph.osd.6:waiting for process to exit 2024-03-29T11:30:29.971 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:30:30.041 INFO:tasks.ceph.osd.6:Stopped 2024-03-29T11:30:30.041 DEBUG:tasks.ceph.osd.7:waiting for process to exit 2024-03-29T11:30:30.041 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:30:30.121 INFO:tasks.ceph.osd.7:Stopped 2024-03-29T11:30:30.121 INFO:teuthology.misc:Shutting down mgr daemons... 2024-03-29T11:30:30.122 DEBUG:tasks.ceph.mgr.y:waiting for process to exit 2024-03-29T11:30:30.122 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:30:30.145 INFO:tasks.ceph.mgr.y:Stopped 2024-03-29T11:30:30.145 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2024-03-29T11:30:30.145 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:30:30.188 INFO:tasks.ceph.mgr.x:Stopped 2024-03-29T11:30:30.188 INFO:teuthology.misc:Shutting down mon daemons... 2024-03-29T11:30:30.188 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-03-29T11:30:30.189 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:30:30.200 INFO:tasks.ceph.mon.a:Stopped 2024-03-29T11:30:30.201 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-03-29T11:30:30.201 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:30:30.252 INFO:tasks.ceph.mon.c:Stopped 2024-03-29T11:30:30.252 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-03-29T11:30:30.252 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:30:30.264 INFO:tasks.ceph.mon.b:Stopped 2024-03-29T11:30:30.264 INFO:tasks.ceph:Checking cluster log for badness... 2024-03-29T11:30:30.265 DEBUG:teuthology.orchestra.run.smithi084:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(OSD_SLOW_PING_TIME' | egrep -v 'reached quota' | egrep -v 'overall HEALTH_' | egrep -v '\(CACHE_POOL_NO_HIT_SET\)' | egrep -v '\(POOL_FULL\)' | egrep -v '\(SMALLER_PGP_NUM\)' | egrep -v '\(SLOW_OPS\)' | egrep -v '\(CACHE_POOL_NEAR_FULL\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v '\(PG_AVAILABILITY\)' | egrep -v '\(OBJECT_MISPLACED\)' | egrep -v 'slow request' | head -n 1 2024-03-29T11:30:30.327 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi084.front.sepia.ceph.com 2024-03-29T11:30:30.327 DEBUG:teuthology.orchestra.run.smithi084:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2024-03-29T11:30:33.512 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi084.front.sepia.ceph.com 2024-03-29T11:30:33.512 DEBUG:teuthology.orchestra.run.smithi084:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2024-03-29T11:30:33.605 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi084.front.sepia.ceph.com 2024-03-29T11:30:33.605 DEBUG:teuthology.orchestra.run.smithi084:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2024-03-29T11:30:33.672 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-3 on ubuntu@smithi084.front.sepia.ceph.com 2024-03-29T11:30:33.672 DEBUG:teuthology.orchestra.run.smithi084:> sync && sudo umount -f /var/lib/ceph/osd/ceph-3 2024-03-29T11:30:33.731 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-4 on ubuntu@smithi179.front.sepia.ceph.com 2024-03-29T11:30:33.731 DEBUG:teuthology.orchestra.run.smithi179:> sync && sudo umount -f /var/lib/ceph/osd/ceph-4 2024-03-29T11:30:33.869 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-5 on ubuntu@smithi179.front.sepia.ceph.com 2024-03-29T11:30:33.869 DEBUG:teuthology.orchestra.run.smithi179:> sync && sudo umount -f /var/lib/ceph/osd/ceph-5 2024-03-29T11:30:33.954 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-6 on ubuntu@smithi179.front.sepia.ceph.com 2024-03-29T11:30:33.954 DEBUG:teuthology.orchestra.run.smithi179:> sync && sudo umount -f /var/lib/ceph/osd/ceph-6 2024-03-29T11:30:34.011 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-7 on ubuntu@smithi179.front.sepia.ceph.com 2024-03-29T11:30:34.012 DEBUG:teuthology.orchestra.run.smithi179:> sync && sudo umount -f /var/lib/ceph/osd/ceph-7 2024-03-29T11:30:34.062 INFO:tasks.ceph:Archiving mon data... 2024-03-29T11:30:34.064 DEBUG:teuthology.misc:Transferring archived files from smithi084:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622699/data/mon.a.tgz 2024-03-29T11:30:34.064 DEBUG:teuthology.orchestra.run.smithi084:> mktemp 2024-03-29T11:30:34.083 INFO:teuthology.orchestra.run.smithi084.stdout:/tmp/tmp.GMI4RpRO5F 2024-03-29T11:30:34.083 DEBUG:teuthology.orchestra.run.smithi084:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.GMI4RpRO5F 2024-03-29T11:30:34.190 DEBUG:teuthology.orchestra.run.smithi084:> sudo chmod 0666 /tmp/tmp.GMI4RpRO5F 2024-03-29T11:30:34.279 DEBUG:teuthology.orchestra.remote:smithi084:/tmp/tmp.GMI4RpRO5F is 126KB 2024-03-29T11:30:34.331 DEBUG:teuthology.orchestra.run.smithi084:> rm -fr /tmp/tmp.GMI4RpRO5F 2024-03-29T11:30:34.347 DEBUG:teuthology.misc:Transferring archived files from smithi084:/var/lib/ceph/mon/ceph-c to /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622699/data/mon.c.tgz 2024-03-29T11:30:34.347 DEBUG:teuthology.orchestra.run.smithi084:> mktemp 2024-03-29T11:30:34.402 INFO:teuthology.orchestra.run.smithi084.stdout:/tmp/tmp.99zblewGOH 2024-03-29T11:30:34.402 DEBUG:teuthology.orchestra.run.smithi084:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-c -- . > /tmp/tmp.99zblewGOH 2024-03-29T11:30:34.490 DEBUG:teuthology.orchestra.run.smithi084:> sudo chmod 0666 /tmp/tmp.99zblewGOH 2024-03-29T11:30:34.576 DEBUG:teuthology.orchestra.remote:smithi084:/tmp/tmp.99zblewGOH is 126KB 2024-03-29T11:30:34.626 DEBUG:teuthology.orchestra.run.smithi084:> rm -fr /tmp/tmp.99zblewGOH 2024-03-29T11:30:34.641 DEBUG:teuthology.misc:Transferring archived files from smithi179:/var/lib/ceph/mon/ceph-b to /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622699/data/mon.b.tgz 2024-03-29T11:30:34.641 DEBUG:teuthology.orchestra.run.smithi179:> mktemp 2024-03-29T11:30:34.659 INFO:teuthology.orchestra.run.smithi179.stdout:/tmp/tmp.YPPqmr740o 2024-03-29T11:30:34.659 DEBUG:teuthology.orchestra.run.smithi179:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-b -- . > /tmp/tmp.YPPqmr740o 2024-03-29T11:30:34.762 DEBUG:teuthology.orchestra.run.smithi179:> sudo chmod 0666 /tmp/tmp.YPPqmr740o 2024-03-29T11:30:34.850 DEBUG:teuthology.orchestra.remote:smithi179:/tmp/tmp.YPPqmr740o is 126KB 2024-03-29T11:30:34.902 DEBUG:teuthology.orchestra.run.smithi179:> rm -fr /tmp/tmp.YPPqmr740o 2024-03-29T11:30:34.917 INFO:tasks.ceph:Cleaning ceph cluster... 2024-03-29T11:30:34.918 DEBUG:teuthology.orchestra.run.smithi084:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-03-29T11:30:34.920 DEBUG:teuthology.orchestra.run.smithi179:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-03-29T11:30:35.000 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-03-29T11:30:35.041 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-03-29T11:30:35.041 INFO:tasks.ceph:Archiving crash dumps... 2024-03-29T11:30:35.043 DEBUG:teuthology.misc:Transferring archived files from smithi084:/var/lib/ceph/crash to /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622699/remote/smithi084/crash 2024-03-29T11:30:35.044 DEBUG:teuthology.orchestra.run.smithi084:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-03-29T11:30:35.071 DEBUG:teuthology.misc:Transferring archived files from smithi179:/var/lib/ceph/crash to /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622699/remote/smithi179/crash 2024-03-29T11:30:35.072 DEBUG:teuthology.orchestra.run.smithi179:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-03-29T11:30:35.109 INFO:tasks.ceph:Compressing logs... 2024-03-29T11:30:35.109 DEBUG:teuthology.orchestra.run.smithi084:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-29T11:30:35.116 DEBUG:teuthology.orchestra.run.smithi179:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-29T11:30:35.180 INFO:teuthology.orchestra.run.smithi084.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38348.log 2024-03-29T11:30:35.180 INFO:teuthology.orchestra.run.smithi084.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39947.log 2024-03-29T11:30:35.181 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.38348.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.a.log 2024-03-29T11:30:35.181 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.39947.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39752.log 2024-03-29T11:30:35.181 INFO:teuthology.orchestra.run.smithi084.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.38348.log.gz 2024-03-29T11:30:35.181 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41173.log 2024-03-29T11:30:35.181 INFO:teuthology.orchestra.run.smithi084.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.39947.log.gz 2024-03-29T11:30:35.181 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.39752.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40379.log 2024-03-29T11:30:35.181 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.41173.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40223.log 2024-03-29T11:30:35.181 INFO:teuthology.orchestra.run.smithi084.stderr: 85.8% -- replaced with /var/log/ceph/ceph-client.admin.39752.log.gz 2024-03-29T11:30:35.181 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.40379.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38660.log 2024-03-29T11:30:35.181 INFO:teuthology.orchestra.run.smithi084.stderr: 85.6% -- replaced with /var/log/ceph/ceph-client.admin.41173.log.gz 2024-03-29T11:30:35.181 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.40223.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38192.log 2024-03-29T11:30:35.181 INFO:teuthology.orchestra.run.smithi084.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.40379.log.gz 2024-03-29T11:30:35.181 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.38660.log: 85.9% -- replaced with /var/log/ceph/ceph-client.admin.40223.log.gz 2024-03-29T11:30:35.181 INFO:teuthology.orchestra.run.smithi084.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38972.log 2024-03-29T11:30:35.181 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.38192.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34689.log 2024-03-29T11:30:35.182 INFO:teuthology.orchestra.run.smithi084.stderr: 85.7% -- replaced with /var/log/ceph/ceph-client.admin.38660.log.gz 2024-03-29T11:30:35.182 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.38972.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40340.log 2024-03-29T11:30:35.182 INFO:teuthology.orchestra.run.smithi084.stderr: 85.8% -- replaced with /var/log/ceph/ceph-client.admin.38192.log.gz 2024-03-29T11:30:35.182 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.34689.log: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.38972.log.gz 2024-03-29T11:30:35.182 INFO:teuthology.orchestra.run.smithi084.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38621.log 2024-03-29T11:30:35.182 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.40340.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40900.log 2024-03-29T11:30:35.182 INFO:teuthology.orchestra.run.smithi084.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.34689.log.gz 2024-03-29T11:30:35.182 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.38621.log: gzip 85.8% -5 --verbose -- replaced with /var/log/ceph/ceph-client.admin.40340.log.gz 2024-03-29T11:30:35.182 INFO:teuthology.orchestra.run.smithi084.stderr: -- /var/log/ceph/ceph-client.admin.39869.log 2024-03-29T11:30:35.182 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.40900.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41329.log 2024-03-29T11:30:35.182 INFO:teuthology.orchestra.run.smithi084.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.38621.log.gz 2024-03-29T11:30:35.182 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.39869.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39557.log 2024-03-29T11:30:35.182 INFO:teuthology.orchestra.run.smithi084.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.40900.log.gz 2024-03-29T11:30:35.182 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.41329.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.39869.log.gz 2024-03-29T11:30:35.183 INFO:teuthology.orchestra.run.smithi084.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40496.log 2024-03-29T11:30:35.183 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.39557.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39908.log 2024-03-29T11:30:35.183 INFO:teuthology.orchestra.run.smithi084.stderr: 86.0% -- replaced with /var/log/ceph/ceph-client.admin.41329.log.gz 2024-03-29T11:30:35.183 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.40496.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41134.log 2024-03-29T11:30:35.183 INFO:teuthology.orchestra.run.smithi084.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.39557.log.gz 2024-03-29T11:30:35.183 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.39908.log: 86.5%gzip -- replaced with /var/log/ceph/ceph-client.admin.40496.log.gz 2024-03-29T11:30:35.183 INFO:teuthology.orchestra.run.smithi084.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.37954.log 2024-03-29T11:30:35.183 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.41134.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38582.log 2024-03-29T11:30:35.183 INFO:teuthology.orchestra.run.smithi084.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.39908.log.gz 2024-03-29T11:30:35.183 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.37954.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39440.log 2024-03-29T11:30:35.183 INFO:teuthology.orchestra.run.smithi084.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.41134.log.gz 2024-03-29T11:30:35.183 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.38582.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38033.log 2024-03-29T11:30:35.183 INFO:teuthology.orchestra.run.smithi084.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.37954.log.gz 2024-03-29T11:30:35.183 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.39440.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40106.log 2024-03-29T11:30:35.184 INFO:teuthology.orchestra.run.smithi084.stderr: 85.8% -- replaced with /var/log/ceph/ceph-client.admin.38582.log.gz 2024-03-29T11:30:35.184 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.38033.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40861.log 2024-03-29T11:30:35.184 INFO:teuthology.orchestra.run.smithi084.stderr: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.39440.log.gz 2024-03-29T11:30:35.184 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.40106.log: 85.9% -- replaced with /var/log/ceph/ceph-client.admin.38033.log.gz 2024-03-29T11:30:35.184 INFO:teuthology.orchestra.run.smithi084.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40145.log 2024-03-29T11:30:35.184 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.40861.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39089.log 2024-03-29T11:30:35.184 INFO:teuthology.orchestra.run.smithi084.stderr: 86.0% -- replaced with /var/log/ceph/ceph-client.admin.40106.log.gz 2024-03-29T11:30:35.184 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.40145.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39401.log 2024-03-29T11:30:35.184 INFO:teuthology.orchestra.run.smithi084.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.40861.log.gz 2024-03-29T11:30:35.184 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.39089.log: 85.9% -- replaced with /var/log/ceph/ceph-client.admin.40145.log.gz 2024-03-29T11:30:35.184 INFO:teuthology.orchestra.run.smithi084.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38309.log 2024-03-29T11:30:35.184 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.39401.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39518.log 2024-03-29T11:30:35.184 INFO:teuthology.orchestra.run.smithi084.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.39089.log.gz 2024-03-29T11:30:35.184 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.38309.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41017.log 2024-03-29T11:30:35.184 INFO:teuthology.orchestra.run.smithi084.stderr: 86.0% -- replaced with /var/log/ceph/ceph-client.admin.39401.log.gz 2024-03-29T11:30:35.185 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.39518.log: 86.0% -- replaced with /var/log/ceph/ceph-client.admin.38309.log.gz 2024-03-29T11:30:35.185 INFO:teuthology.orchestra.run.smithi084.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39830.log 2024-03-29T11:30:35.185 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.41017.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-03-29T11:30:35.185 INFO:teuthology.orchestra.run.smithi084.stderr: 85.7% -- replaced with /var/log/ceph/ceph-client.admin.39518.log.gz 2024-03-29T11:30:35.185 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.39830.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38933.log 2024-03-29T11:30:35.185 INFO:teuthology.orchestra.run.smithi084.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.41017.log.gz 2024-03-29T11:30:35.185 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph.audit.log: 85.8% -- replaced with /var/log/ceph/ceph-client.admin.39830.log.gz 2024-03-29T11:30:35.185 INFO:teuthology.orchestra.run.smithi084.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40301.log 2024-03-29T11:30:35.185 INFO:teuthology.orchestra.run.smithi084.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40586.log 2024-03-29T11:30:35.185 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.38933.log: /var/log/ceph/ceph-client.admin.40301.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39245.log 2024-03-29T11:30:35.185 INFO:teuthology.orchestra.run.smithi084.stderr: 93.5% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-03-29T11:30:35.185 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.40586.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41056.log 2024-03-29T11:30:35.185 INFO:teuthology.orchestra.run.smithi084.stderr: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.38933.log.gz 2024-03-29T11:30:35.185 INFO:teuthology.orchestra.run.smithi084.stderr: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.40301.log.gz 2024-03-29T11:30:35.185 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.39245.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39986.log 2024-03-29T11:30:35.186 INFO:teuthology.orchestra.run.smithi084.stderr: 85.5% -- replaced with /var/log/ceph/ceph-client.admin.40586.log.gz 2024-03-29T11:30:35.186 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.41056.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40939.log 2024-03-29T11:30:35.186 INFO:teuthology.orchestra.run.smithi084.stderr: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.39245.log.gz 2024-03-29T11:30:35.186 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.39986.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41368.log 2024-03-29T11:30:35.186 INFO:teuthology.orchestra.run.smithi084.stderr: 85.7% -- replaced with /var/log/ceph/ceph-client.admin.41056.log.gz 2024-03-29T11:30:35.186 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.40939.log: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.39986.log.gz 2024-03-29T11:30:35.186 INFO:teuthology.orchestra.run.smithi084.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39323.log 2024-03-29T11:30:35.186 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.41368.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41212.log 2024-03-29T11:30:35.186 INFO:teuthology.orchestra.run.smithi084.stderr: 86.0% -- replaced with /var/log/ceph/ceph-client.admin.40939.log.gz 2024-03-29T11:30:35.186 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.39323.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.1.log 2024-03-29T11:30:35.186 INFO:teuthology.orchestra.run.smithi084.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.41368.log.gz 2024-03-29T11:30:35.186 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.41212.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39128.log 2024-03-29T11:30:35.186 INFO:teuthology.orchestra.run.smithi084.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.39323.log.gz 2024-03-29T11:30:35.186 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40025.log 2024-03-29T11:30:35.187 INFO:teuthology.orchestra.run.smithi084.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.41212.log.gz 2024-03-29T11:30:35.187 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.39128.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38543.log 2024-03-29T11:30:35.187 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.40025.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38387.log 2024-03-29T11:30:35.187 INFO:teuthology.orchestra.run.smithi084.stderr: 85.6% -- replaced with /var/log/ceph/ceph-client.admin.39128.log.gz 2024-03-29T11:30:35.187 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.38543.log: 85.6% -- replaced with /var/log/ceph/ceph-client.admin.40025.log.gz 2024-03-29T11:30:35.187 INFO:teuthology.orchestra.run.smithi084.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39635.log 2024-03-29T11:30:35.187 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.38387.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38894.log 2024-03-29T11:30:35.187 INFO:teuthology.orchestra.run.smithi084.stderr: 86.0% -- replaced with /var/log/ceph/ceph-client.admin.38543.log.gz 2024-03-29T11:30:35.187 INFO:teuthology.orchestra.run.smithi084.stderr: 85.9% -- replaced with /var/log/ceph/ceph-client.admin.38387.log.gz 2024-03-29T11:30:35.187 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.39635.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34586.log 2024-03-29T11:30:35.187 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.38894.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37994.log 2024-03-29T11:30:35.187 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.34586.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.39635.log.gz 2024-03-29T11:30:35.187 INFO:teuthology.orchestra.run.smithi084.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39167.log 2024-03-29T11:30:35.187 INFO:teuthology.orchestra.run.smithi084.stderr: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.38894.log.gz 2024-03-29T11:30:35.188 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.37994.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40744.log 2024-03-29T11:30:35.188 INFO:teuthology.orchestra.run.smithi084.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.34586.log.gz 2024-03-29T11:30:35.188 INFO:teuthology.orchestra.run.smithi084.stderr:gzip -5 --verbose/var/log/ceph/ceph-client.admin.39167.log: -- /var/log/ceph/ceph-client.admin.38111.log 2024-03-29T11:30:35.188 INFO:teuthology.orchestra.run.smithi084.stderr: 85.6% -- replaced with /var/log/ceph/ceph-client.admin.37994.log.gz 2024-03-29T11:30:35.188 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.40744.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39206.log 2024-03-29T11:30:35.188 INFO:teuthology.orchestra.run.smithi084.stderr: 86.0% -- replaced with /var/log/ceph/ceph-client.admin.39167.log.gz 2024-03-29T11:30:35.188 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.38111.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39011.log 2024-03-29T11:30:35.188 INFO:teuthology.orchestra.run.smithi084.stderr: 85.9% -- replaced with /var/log/ceph/ceph-client.admin.40744.log.gz 2024-03-29T11:30:35.188 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.39206.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40184.log 2024-03-29T11:30:35.188 INFO:teuthology.orchestra.run.smithi084.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.38111.log.gz 2024-03-29T11:30:35.188 INFO:teuthology.orchestra.run.smithi084.stderr:gzip -5/var/log/ceph/ceph-client.admin.39011.log: --verbose -- /var/log/ceph/ceph-osd.3.log 2024-03-29T11:30:35.188 INFO:teuthology.orchestra.run.smithi084.stderr: 86.0% -- replaced with /var/log/ceph/ceph-client.admin.39206.log.gz 2024-03-29T11:30:35.188 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.40184.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38465.log 2024-03-29T11:30:35.188 INFO:teuthology.orchestra.run.smithi084.stderr: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.39011.log.gz 2024-03-29T11:30:35.188 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-osd.3.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38816.log 2024-03-29T11:30:35.189 INFO:teuthology.orchestra.run.smithi084.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.40184.log.gz 2024-03-29T11:30:35.189 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.38465.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39362.log 2024-03-29T11:30:35.189 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.38816.log: gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.28476.log 2024-03-29T11:30:35.189 INFO:teuthology.orchestra.run.smithi084.stderr: 85.9% -- replaced with /var/log/ceph/ceph-client.admin.38465.log.gz 2024-03-29T11:30:35.189 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.39362.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40418.log 2024-03-29T11:30:35.189 INFO:teuthology.orchestra.run.smithi084.stderr: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.38816.log.gz 2024-03-29T11:30:35.189 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph.tmp-client.admin.28476.log: gzip 0.0% -5 -- replaced with /var/log/ceph/ceph.tmp-client.admin.28476.log.gz 2024-03-29T11:30:35.189 INFO:teuthology.orchestra.run.smithi084.stderr: --verbose -- /var/log/ceph/ceph-client.admin.39713.log 2024-03-29T11:30:35.189 INFO:teuthology.orchestra.run.smithi084.stderr: 86.0% -- replaced with /var/log/ceph/ceph-client.admin.39362.log.gz 2024-03-29T11:30:35.189 INFO:teuthology.orchestra.run.smithi084.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38270.log 2024-03-29T11:30:35.189 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.40418.log: /var/log/ceph/ceph-client.admin.39713.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39596.log 2024-03-29T11:30:35.189 INFO:teuthology.orchestra.run.smithi084.stderr: 85.7% -- replaced with /var/log/ceph/ceph-client.admin.40418.log.gz 2024-03-29T11:30:35.189 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.38270.log: gzip -5 86.6% --verbose -- -- replaced with /var/log/ceph/ceph-client.admin.39713.log.gz /var/log/ceph/ceph-client.admin.38777.log 2024-03-29T11:30:35.189 INFO:teuthology.orchestra.run.smithi084.stderr: 2024-03-29T11:30:35.189 INFO:teuthology.orchestra.run.smithi084.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.38270.log.gz 2024-03-29T11:30:35.190 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.39596.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39284.log 2024-03-29T11:30:35.190 INFO:teuthology.orchestra.run.smithi084.stderr:gzip/var/log/ceph/ceph-client.admin.38777.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.38699.log 2024-03-29T11:30:35.190 INFO:teuthology.orchestra.run.smithi084.stderr: 85.8% -- replaced with /var/log/ceph/ceph-client.admin.39596.log.gz 2024-03-29T11:30:35.190 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.39284.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34295.log 2024-03-29T11:30:35.190 INFO:teuthology.orchestra.run.smithi084.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.38777.log.gz 2024-03-29T11:30:35.190 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.38699.log: 86.0% -- replaced with /var/log/ceph/ceph-client.admin.39284.log.gz 2024-03-29T11:30:35.190 INFO:teuthology.orchestra.run.smithi084.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38426.log 2024-03-29T11:30:35.190 INFO:teuthology.orchestra.run.smithi084.stderr: 86.0% -- replaced with /var/log/ceph/ceph-client.admin.38699.log.gz 2024-03-29T11:30:35.190 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.34295.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.y.log 2024-03-29T11:30:35.190 INFO:teuthology.orchestra.run.smithi084.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41095.log 2024-03-29T11:30:35.190 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.38426.log: /var/log/ceph/ceph-mgr.y.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.34295.log.gz 2024-03-29T11:30:35.190 INFO:teuthology.orchestra.run.smithi084.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40822.log 2024-03-29T11:30:35.190 INFO:teuthology.orchestra.run.smithi084.stderr: 85.7% -- replaced with /var/log/ceph/ceph-client.admin.38426.log.gz 2024-03-29T11:30:35.190 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.41095.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40666.log 2024-03-29T11:30:35.191 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.40822.log: gzip 86.1% -- replaced with /var/log/ceph/ceph-client.admin.41095.log.gz 2024-03-29T11:30:35.191 INFO:teuthology.orchestra.run.smithi084.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.39479.log 2024-03-29T11:30:35.191 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.40666.log: gzip 86.3% -- replaced with /var/log/ceph/ceph-client.admin.40822.log.gz 2024-03-29T11:30:35.191 INFO:teuthology.orchestra.run.smithi084.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.40457.log 2024-03-29T11:30:35.191 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.39479.log: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.40666.log.gz 2024-03-29T11:30:35.191 INFO:teuthology.orchestra.run.smithi084.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41251.log 2024-03-29T11:30:35.191 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.40457.log: 85.5%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38504.log 2024-03-29T11:30:35.191 INFO:teuthology.orchestra.run.smithi084.stderr: -- replaced with /var/log/ceph/ceph-client.admin.39479.log.gz 2024-03-29T11:30:35.191 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.41251.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39674.log 2024-03-29T11:30:35.191 INFO:teuthology.orchestra.run.smithi084.stderr: 85.6% -- replaced with /var/log/ceph/ceph-client.admin.40457.log.gz 2024-03-29T11:30:35.191 INFO:teuthology.orchestra.run.smithi084.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.0.log 2024-03-29T11:30:35.191 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.38504.log: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.41251.log.gz 2024-03-29T11:30:35.191 INFO:teuthology.orchestra.run.smithi084.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40978.log 2024-03-29T11:30:35.191 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.39674.log: 86.0% -- replaced with /var/log/ceph/ceph-client.admin.38504.log.gz 2024-03-29T11:30:35.191 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40064.log 2024-03-29T11:30:35.192 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.40978.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38855.log 2024-03-29T11:30:35.192 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.40064.log: 85.9% -- replaced with /var/log/ceph/ceph-client.admin.39674.log.gz 2024-03-29T11:30:35.192 INFO:teuthology.orchestra.run.smithi084.stderr: 85.8% -- replaced with /var/log/ceph/ceph-client.admin.40978.log.gz 2024-03-29T11:30:35.192 INFO:teuthology.orchestra.run.smithi084.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38738.log 2024-03-29T11:30:35.192 INFO:teuthology.orchestra.run.smithi084.stderr: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.40064.log.gz 2024-03-29T11:30:35.192 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.38855.log: gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-03-29T11:30:35.192 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.38738.log: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.38855.log.gz 2024-03-29T11:30:35.192 INFO:teuthology.orchestra.run.smithi084.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40705.log 2024-03-29T11:30:35.192 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph.log: 85.4% -- replaced with /var/log/ceph/ceph-client.admin.38738.log.gz 2024-03-29T11:30:35.192 INFO:teuthology.orchestra.run.smithi084.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2024-03-29T11:30:35.192 INFO:teuthology.orchestra.run.smithi084.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41290.log 2024-03-29T11:30:35.192 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-osd.2.log: /var/log/ceph/ceph-client.admin.40705.log: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.40705.log.gz 2024-03-29T11:30:35.192 INFO:teuthology.orchestra.run.smithi084.stderr: 92.7% -- replaced with /var/log/ceph/ceph.log.gz 2024-03-29T11:30:35.192 INFO:teuthology.orchestra.run.smithi084.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37915.log 2024-03-29T11:30:35.192 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.41290.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38150.log 2024-03-29T11:30:35.193 INFO:teuthology.orchestra.run.smithi084.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.41290.log.gz 2024-03-29T11:30:35.193 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.37915.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40783.log 2024-03-29T11:30:35.193 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.38150.log: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.37915.log.gz 2024-03-29T11:30:35.193 INFO:teuthology.orchestra.run.smithi084.stderr: 86.0% -- replaced with /var/log/ceph/ceph-client.admin.38150.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39791.log 2024-03-29T11:30:35.193 INFO:teuthology.orchestra.run.smithi084.stderr: 2024-03-29T11:30:35.193 INFO:teuthology.orchestra.run.smithi084.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38072.log 2024-03-29T11:30:35.193 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.39791.log: 93.8% -- replaced with /var/log/ceph/ceph-mgr.y.log.gz 2024-03-29T11:30:35.193 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.40783.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39050.log 2024-03-29T11:30:35.193 INFO:teuthology.orchestra.run.smithi084.stderr: 85.7% 85.9% -- replaced with /var/log/ceph/ceph-client.admin.40783.log.gz 2024-03-29T11:30:35.193 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.38072.log: -- replaced with /var/log/ceph/ceph-client.admin.39791.log.gz 2024-03-29T11:30:35.193 INFO:teuthology.orchestra.run.smithi084.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40627.log 2024-03-29T11:30:35.193 INFO:teuthology.orchestra.run.smithi084.stderr: 85.7% -- replaced with /var/log/ceph/ceph-client.admin.38072.log.gz 2024-03-29T11:30:35.193 INFO:teuthology.orchestra.run.smithi084.stderr:gzip -5/var/log/ceph/ceph-client.admin.39050.log: --verbose -- /var/log/ceph/ceph-client.admin.40262.log 2024-03-29T11:30:35.193 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.40627.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38231.log 2024-03-29T11:30:35.194 INFO:teuthology.orchestra.run.smithi084.stderr: 85.4% -- replaced with /var/log/ceph/ceph-client.admin.39050.log.gz 2024-03-29T11:30:35.194 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.40262.log: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.40627.log.gz 2024-03-29T11:30:35.194 INFO:teuthology.orchestra.run.smithi084.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.c.log 2024-03-29T11:30:35.194 INFO:teuthology.orchestra.run.smithi084.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.40262.log.gz 2024-03-29T11:30:35.194 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.38231.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40547.log 2024-03-29T11:30:35.194 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-mon.c.log: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.38231.log.gz 2024-03-29T11:30:35.194 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.40547.log: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.40547.log.gz 2024-03-29T11:30:35.200 INFO:teuthology.orchestra.run.smithi179.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.6.log 2024-03-29T11:30:35.200 INFO:teuthology.orchestra.run.smithi179.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34483.log 2024-03-29T11:30:35.200 INFO:teuthology.orchestra.run.smithi179.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-03-29T11:30:35.200 INFO:teuthology.orchestra.run.smithi179.stderr:/var/log/ceph/ceph-osd.6.log: /var/log/ceph/ceph-client.admin.34483.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34603.log 2024-03-29T11:30:35.200 INFO:teuthology.orchestra.run.smithi179.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.4.log 2024-03-29T11:30:35.200 INFO:teuthology.orchestra.run.smithi179.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.34483.log.gz 2024-03-29T11:30:35.200 INFO:teuthology.orchestra.run.smithi179.stderr:/var/log/ceph/ceph-client.admin.34603.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34763.log 2024-03-29T11:30:35.201 INFO:teuthology.orchestra.run.smithi179.stderr: 88.2% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-03-29T11:30:35.201 INFO:teuthology.orchestra.run.smithi179.stderr:/var/log/ceph/ceph-osd.4.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34643.log 2024-03-29T11:30:35.201 INFO:teuthology.orchestra.run.smithi179.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.34603.log.gz 2024-03-29T11:30:35.201 INFO:teuthology.orchestra.run.smithi179.stderr:/var/log/ceph/ceph-client.admin.34763.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.7.log 2024-03-29T11:30:35.201 INFO:teuthology.orchestra.run.smithi179.stderr:/var/log/ceph/ceph-client.admin.34643.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.5.log 2024-03-29T11:30:35.201 INFO:teuthology.orchestra.run.smithi179.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.34763.log.gz 2024-03-29T11:30:35.201 INFO:teuthology.orchestra.run.smithi179.stderr:/var/log/ceph/ceph-osd.7.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34563.log 2024-03-29T11:30:35.201 INFO:teuthology.orchestra.run.smithi179.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.34643.log.gz 2024-03-29T11:30:35.201 INFO:teuthology.orchestra.run.smithi179.stderr:/var/log/ceph/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34683.log 2024-03-29T11:30:35.201 INFO:teuthology.orchestra.run.smithi179.stderr:/var/log/ceph/ceph-client.admin.34563.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34523.log 2024-03-29T11:30:35.201 INFO:teuthology.orchestra.run.smithi179.stderr:/var/log/ceph/ceph-client.admin.34683.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2024-03-29T11:30:35.201 INFO:teuthology.orchestra.run.smithi179.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.34563.log.gz 2024-03-29T11:30:35.201 INFO:teuthology.orchestra.run.smithi179.stderr:/var/log/ceph/ceph-client.admin.34523.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.34683.log.gz 2024-03-29T11:30:35.201 INFO:teuthology.orchestra.run.smithi179.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34723.log 2024-03-29T11:30:35.201 INFO:teuthology.orchestra.run.smithi179.stderr:/var/log/ceph/ceph-mgr.x.log: gzip -5 87.9% --verbose -- /var/log/ceph/ceph.log 2024-03-29T11:30:35.202 INFO:teuthology.orchestra.run.smithi179.stderr: -- replaced with /var/log/ceph/ceph-client.admin.34523.log.gz 2024-03-29T11:30:35.202 INFO:teuthology.orchestra.run.smithi179.stderr:/var/log/ceph/ceph-client.admin.34723.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.b.log 2024-03-29T11:30:35.202 INFO:teuthology.orchestra.run.smithi179.stderr:/var/log/ceph/ceph.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.34723.log.gz 2024-03-29T11:30:35.202 INFO:teuthology.orchestra.run.smithi179.stderr:/var/log/ceph/ceph-mon.b.log: 87.1% -- replaced with /var/log/ceph/ceph.log.gz 2024-03-29T11:30:35.231 INFO:teuthology.orchestra.run.smithi179.stderr: 92.3% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2024-03-29T11:30:47.161 INFO:teuthology.orchestra.run.smithi084.stderr: 94.6% -- replaced with /var/log/ceph/ceph-osd.3.log.gz 2024-03-29T11:30:47.175 INFO:teuthology.orchestra.run.smithi179.stderr: 94.7% -- replaced with /var/log/ceph/ceph-osd.4.log.gz 2024-03-29T11:30:47.345 INFO:teuthology.orchestra.run.smithi179.stderr: 94.7% -- replaced with /var/log/ceph/ceph-osd.5.log.gz 2024-03-29T11:30:51.179 INFO:teuthology.orchestra.run.smithi179.stderr: 94.7% -- replaced with /var/log/ceph/ceph-osd.7.log.gz 2024-03-29T11:30:51.366 INFO:teuthology.orchestra.run.smithi084.stderr: 94.6% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2024-03-29T11:30:51.515 INFO:teuthology.orchestra.run.smithi179.stderr: 94.7% -- replaced with /var/log/ceph/ceph-osd.6.log.gz 2024-03-29T11:30:52.724 INFO:teuthology.orchestra.run.smithi084.stderr: 94.6% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2024-03-29T11:30:52.800 INFO:teuthology.orchestra.run.smithi084.stderr: 94.6% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2024-03-29T11:31:26.046 INFO:teuthology.orchestra.run.smithi084.stderr: 92.9% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2024-03-29T11:31:28.246 INFO:teuthology.orchestra.run.smithi084.stderr: 92.8% -- replaced with /var/log/ceph/ceph-mon.c.log.gz 2024-03-29T11:31:28.248 INFO:teuthology.orchestra.run.smithi084.stderr: 2024-03-29T11:31:28.248 INFO:teuthology.orchestra.run.smithi084.stderr:real 0m53.119s 2024-03-29T11:31:28.248 INFO:teuthology.orchestra.run.smithi084.stderr:user 2m41.472s 2024-03-29T11:31:28.248 INFO:teuthology.orchestra.run.smithi084.stderr:sys 0m5.621s 2024-03-29T11:31:39.493 INFO:teuthology.orchestra.run.smithi179.stderr: 92.7% -- replaced with /var/log/ceph/ceph-mon.b.log.gz 2024-03-29T11:31:39.495 INFO:teuthology.orchestra.run.smithi179.stderr: 2024-03-29T11:31:39.495 INFO:teuthology.orchestra.run.smithi179.stderr:real 1m4.327s 2024-03-29T11:31:39.495 INFO:teuthology.orchestra.run.smithi179.stderr:user 1m56.232s 2024-03-29T11:31:39.495 INFO:teuthology.orchestra.run.smithi179.stderr:sys 0m4.413s 2024-03-29T11:31:39.495 INFO:tasks.ceph:Archiving logs... 2024-03-29T11:31:39.495 DEBUG:teuthology.misc:Transferring archived files from smithi084:/var/log/ceph to /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622699/remote/smithi084/log 2024-03-29T11:31:39.497 DEBUG:teuthology.orchestra.run.smithi084:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-29T11:32:05.160 DEBUG:teuthology.misc:Transferring archived files from smithi179:/var/log/ceph to /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622699/remote/smithi179/log 2024-03-29T11:32:05.161 DEBUG:teuthology.orchestra.run.smithi179:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-29T11:32:24.165 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1913, in task with contextutil.nested(*subtasks): File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 252, in ceph_log yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds 2024-03-29T11:32:24.371 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=0b60711e9ac14fa3891dfb1540451964 Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1913, in task with contextutil.nested(*subtasks): File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 252, in ceph_log yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds 2024-03-29T11:32:24.374 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-03-29T11:32:24.385 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-03-29T11:32:24.407 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-03-29T11:32:24.407 DEBUG:teuthology.orchestra.run.smithi084:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-29T11:32:24.421 DEBUG:teuthology.orchestra.run.smithi179:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-29T11:32:24.492 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-03-29T11:32:24.501 INFO:teuthology.task.clock:Checking final clock skew... 2024-03-29T11:32:24.502 DEBUG:teuthology.orchestra.run.smithi084:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-29T11:32:24.505 DEBUG:teuthology.orchestra.run.smithi179:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-29T11:32:24.519 INFO:teuthology.orchestra.run.smithi084.stderr:bash: line 1: ntpq: command not found 2024-03-29T11:32:24.519 INFO:teuthology.orchestra.run.smithi179.stderr:bash: line 1: ntpq: command not found 2024-03-29T11:32:24.522 INFO:teuthology.orchestra.run.smithi084.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-29T11:32:24.523 INFO:teuthology.orchestra.run.smithi084.stdout:=============================================================================== 2024-03-29T11:32:24.523 INFO:teuthology.orchestra.run.smithi084.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 22 +462us[ +449us] +/- 92ms 2024-03-29T11:32:24.523 INFO:teuthology.orchestra.run.smithi084.stdout:^* hv02.front.sepia.ceph.com 3 6 377 20 -104us[ -118us] +/- 55ms 2024-03-29T11:32:24.523 INFO:teuthology.orchestra.run.smithi084.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 16 -253us[ -253us] +/- 86ms 2024-03-29T11:32:24.523 INFO:teuthology.orchestra.run.smithi084.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T11:32:24.525 INFO:teuthology.orchestra.run.smithi179.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-29T11:32:24.525 INFO:teuthology.orchestra.run.smithi179.stdout:=============================================================================== 2024-03-29T11:32:24.525 INFO:teuthology.orchestra.run.smithi179.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 19 +476us[ +464us] +/- 92ms 2024-03-29T11:32:24.525 INFO:teuthology.orchestra.run.smithi179.stdout:^* hv02.front.sepia.ceph.com 3 6 377 18 -107us[ -120us] +/- 55ms 2024-03-29T11:32:24.526 INFO:teuthology.orchestra.run.smithi179.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 17 -264us[ -264us] +/- 86ms 2024-03-29T11:32:24.526 INFO:teuthology.orchestra.run.smithi179.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T11:32:24.526 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-03-29T11:32:24.535 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-03-29T11:32:24.536 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-03-29T11:32:24.547 DEBUG:teuthology.orchestra.run.smithi084:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-29T11:32:24.635 DEBUG:teuthology.orchestra.run.smithi179:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-29T11:32:24.699 DEBUG:teuthology.orchestra.run.smithi084:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-29T11:32:24.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-29T11:32:24.730 DEBUG:teuthology.orchestra.run.smithi179:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-29T11:32:24.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-29T11:32:24.760 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-03-29T11:32:24.772 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-03-29T11:32:24.783 INFO:teuthology.task.internal:Duration was 1254.598071 seconds 2024-03-29T11:32:24.784 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-03-29T11:32:24.866 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-03-29T11:32:24.867 DEBUG:teuthology.orchestra.run.smithi084:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-29T11:32:24.869 DEBUG:teuthology.orchestra.run.smithi179:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-29T11:32:24.912 INFO:teuthology.orchestra.run.smithi084.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-29T11:32:24.913 INFO:teuthology.orchestra.run.smithi179.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-29T11:32:25.013 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-03-29T11:32:25.014 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi084.front.sepia.ceph.com 2024-03-29T11:32:25.014 DEBUG:teuthology.orchestra.run.smithi084:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-29T11:32:25.033 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi179.front.sepia.ceph.com 2024-03-29T11:32:25.034 DEBUG:teuthology.orchestra.run.smithi179:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-29T11:32:25.057 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-03-29T11:32:25.057 DEBUG:teuthology.orchestra.run.smithi084:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-29T11:32:25.077 DEBUG:teuthology.orchestra.run.smithi179:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-29T11:32:25.125 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-03-29T11:32:25.125 DEBUG:teuthology.orchestra.run.smithi084:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-29T11:32:25.145 DEBUG:teuthology.orchestra.run.smithi179:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-29T11:32:25.262 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-03-29T11:32:25.273 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-03-29T11:32:25.273 DEBUG:teuthology.orchestra.run.smithi084:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-29T11:32:25.300 DEBUG:teuthology.orchestra.run.smithi179:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-29T11:32:25.328 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-03-29T11:32:25.344 DEBUG:teuthology.orchestra.run.smithi084:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-29T11:32:25.347 DEBUG:teuthology.orchestra.run.smithi179:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-29T11:32:25.370 INFO:teuthology.orchestra.run.smithi084.stdout:kernel.core_pattern = core 2024-03-29T11:32:25.394 INFO:teuthology.orchestra.run.smithi179.stdout:kernel.core_pattern = core 2024-03-29T11:32:25.421 DEBUG:teuthology.orchestra.run.smithi084:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-29T11:32:25.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-29T11:32:25.449 DEBUG:teuthology.orchestra.run.smithi179:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-29T11:32:25.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-29T11:32:25.476 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-03-29T11:32:25.492 INFO:teuthology.task.internal:Transferring archived files... 2024-03-29T11:32:25.494 DEBUG:teuthology.misc:Transferring archived files from smithi084:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622699/remote/smithi084 2024-03-29T11:32:25.494 DEBUG:teuthology.orchestra.run.smithi084:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-29T11:32:25.531 DEBUG:teuthology.misc:Transferring archived files from smithi179:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622699/remote/smithi179 2024-03-29T11:32:25.531 DEBUG:teuthology.orchestra.run.smithi179:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-29T11:32:25.565 INFO:teuthology.task.internal:Removing archive directory... 2024-03-29T11:32:25.566 DEBUG:teuthology.orchestra.run.smithi084:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-29T11:32:25.568 DEBUG:teuthology.orchestra.run.smithi179:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-29T11:32:25.615 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-03-29T11:32:25.627 INFO:teuthology.task.internal:Not uploading archives. 2024-03-29T11:32:25.627 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-03-29T11:32:25.639 INFO:teuthology.task.internal:Tidying up after the test... 2024-03-29T11:32:25.639 DEBUG:teuthology.orchestra.run.smithi084:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-29T11:32:25.642 DEBUG:teuthology.orchestra.run.smithi179:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-29T11:32:25.653 INFO:teuthology.orchestra.run.smithi084.stdout: 262166 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 29 11:32 /home/ubuntu/cephtest 2024-03-29T11:32:25.670 INFO:teuthology.orchestra.run.smithi179.stdout: 265844 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 29 11:32 /home/ubuntu/cephtest 2024-03-29T11:32:25.671 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-03-29T11:32:25.682 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-03-29T11:32:25.703 INFO:teuthology.nuke:Checking targets against current locks 2024-03-29T11:32:25.730 DEBUG:teuthology.nuke:shortname: smithi084 2024-03-29T11:32:25.731 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-29T11:32:25.755 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi084.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622699', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-29 11:02:10.786447', 'locked_by': 'scheduled_amathuri@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBEHXNFai/jEmct9ZIBZFZkJRhsklU2T6DhlQYD7OOFnU/7MBtkG1eum5vdqLyZuTMzqVORCX+BgUC7grhC+ksv4='} 2024-03-29T11:32:25.759 DEBUG:teuthology.nuke:shortname: smithi179 2024-03-29T11:32:25.759 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-29T11:32:25.783 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi179.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622699', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-29 11:02:10.788354', 'locked_by': 'scheduled_amathuri@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPL2qn7wt1ESWLkdih7XgUUtWDFzamLA5GFZpkPo13TiiE7jP24czk2rm6vPwLHGZeXugXdNWG/bPvTWDQXA12I='} 2024-03-29T11:32:25.809 INFO:teuthology.orchestra.console.smithi084:Power off 2024-03-29T11:32:25.809 DEBUG:teuthology.orchestra.console.smithi084:pexpect command: ipmitool -H smithi084.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-29T11:32:25.834 DEBUG:teuthology.orchestra.console.smithi084:power off output: Chassis Power Control: Down/Off 2024-03-29T11:32:25.834 DEBUG:teuthology.orchestra.console.smithi084:pexpect command: ipmitool -H smithi084.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-29T11:32:25.852 INFO:teuthology.orchestra.console.smithi179:Power off 2024-03-29T11:32:25.852 DEBUG:teuthology.orchestra.console.smithi179:pexpect command: ipmitool -H smithi179.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-29T11:32:25.863 DEBUG:teuthology.orchestra.console.smithi084:check power output: Chassis Power is on 2024-03-29T11:32:25.875 DEBUG:teuthology.orchestra.console.smithi179:power off output: Chassis Power Control: Down/Off 2024-03-29T11:32:25.875 DEBUG:teuthology.orchestra.console.smithi179:pexpect command: ipmitool -H smithi179.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-29T11:32:25.901 DEBUG:teuthology.orchestra.console.smithi179:check power output: Chassis Power is on 2024-03-29T11:32:29.864 DEBUG:teuthology.orchestra.console.smithi084:pexpect command: ipmitool -H smithi084.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-29T11:32:29.902 DEBUG:teuthology.orchestra.console.smithi179:pexpect command: ipmitool -H smithi179.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-29T11:32:29.978 DEBUG:teuthology.orchestra.console.smithi084:check power output: Chassis Power is on 2024-03-29T11:32:30.017 DEBUG:teuthology.orchestra.console.smithi179:check power output: Chassis Power is on 2024-03-29T11:32:33.983 DEBUG:teuthology.orchestra.console.smithi084:pexpect command: ipmitool -H smithi084.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-29T11:32:34.401 DEBUG:teuthology.orchestra.console.smithi179:pexpect command: ipmitool -H smithi179.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-29T11:32:34.504 DEBUG:teuthology.orchestra.console.smithi084:check power output: Chassis Power is off 2024-03-29T11:32:34.514 DEBUG:teuthology.orchestra.console.smithi179:check power output: Chassis Power is off 2024-03-29T11:32:34.605 INFO:teuthology.orchestra.console.smithi084:Power off completed 2024-03-29T11:32:34.616 INFO:teuthology.orchestra.console.smithi179:Power off completed 2024-03-29T11:32:34.961 INFO:teuthology.run:Summary data: description: rados:verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-snappy rados tasks/rados_api_tests validater/lockdep} duration: 1254.5980713367462 failure_reason: reached maximum tries (91) after waiting for 540 seconds flavor: default owner: scheduled_amathuri@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=0b60711e9ac14fa3891dfb1540451964 status: fail success: false 2024-03-29T11:32:34.961 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-29T11:32:35.040 INFO:teuthology.run:FAIL